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SECTION  1 


INTRODUCTION 


1.1  PURPOSE 

There  has  been  an  increased  awareness  in  recent  years  of  the  critical 
problems  that  have  been  encountered  in  the  development  of  large  scale- 
software  systems.  These  problems  not  only  include  the  cost  and  schedule 
overruns  typical  of  development  efforts,  and  the  poor  performance  of  the 
systems  once  they  are  delivered,  but  also  include  the  high  cost  ofmain- 
taining  the  systems,  the  lack  of  portability,  and  the  high  sensitivity  to 
changes  in  requirements. 

The  government  and  00D  in  particular,  as  customers  of  many  large  scale 
software  system  developments,  have  sponsored  many  research  efforts  aimed 
at  attacking  these  problems.  For  example,  the  efforts  related  to  the 
development  of  a  standard  000  programing  language,  software  development 
techniques,  and  development  tools  and  aids  all  provide  partial  solution  to 
the  above  problems  by  encouraging  a  more  disciplined  approach  to  the  devel¬ 
opment  of  software  and  therefore  a  more  controlled  development  process. 

A  related  research  thrust  which  has  been  recently  funded  by  DOD  is  the 
area  of  software  metrics.  The  research  in  this  area  has  resulted  in  the 
development  and  evaluation  of  a  number  of  metrics  which  measure  various 
attributes  of  software  and  relate  to  different  aspects  of  software  quality. 

The  potential  of  the  software  metric  concepts  can  be  realized  by  their 
inclusion  in  software  quality  assurance  programs.  Their  Impact  on  a 
quality  assurance  program  is  to  provide  a  more  disciplined,  engineering 
approach  to  quality  assurance  and  to  provide  a  mechanism  for  taking  a 
life  cycle  viewpoint  of  software  quality;  The  benefits  derived  from  their 
application  are  realized  In  life  cycle  cost  reduction. 

The  purpose  of  this  manual  Is  to  present  a  complete  set  of  procedures  and 
guidelines  for  Introducing  and  utilizing  current  software  quality  measure¬ 
ment  techniques  in  a  quality  assurance  program  associated  with  large  scale 
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software  system  developments.  These  procedures  and  guidelines  will  Identify 


1.  How  to  Identify  and  specify  software  quality  requirements  (Setting 
Quality  Goals). 

2.  How  and  when  to  apply  software  metrics  (Applying  Metrics),  and 

3.  How  to  interpret  the  information  obtained  from  the  application  of 
the  metrics  (Making  a  Quality  Assessment). 

1.2  SCOPE 

This  manual  is  based  on  the  results  of  research  conducted  in  support  of 
the  United  States  Air  Force  Electronic  Systems  Division  (ESD),  Rome  Air 
Development  Center  (RADC),  and  the  United  States  Army  Computer  Systems 
Command's  Army  Institute  for  Research  in  Management  Information  and  Computer 
Science  (USACSC/AIRMICS) .  While  aspects  of  the  technology  of  software 
metrics  require  further  research,  those  portions  which  can  currently 
provide  benefit  to  a  software  quality  assurance  program  are  emphasized 
In  this  manual.  Guidelines  and  procedures  for  using  the  software 
metrics  are  described.  The  guidelines  and  procedures  are  presented  In 
such  a  way  as  to  facilitate  their  application  using  this  manual  In  a 
software  development.  All  of  the  procedures  are  described  as  manual 
processes,  however,  where  automated  software  tools  could  be  used  to 
compliment  or  enhance  the  process,  the  tools  are  Identified. 

1.3  QUALITY  MEASUREMENT  IN  PERSPECTIVE 

The  evolution  during  the  past  decade  of  modern  programming  practices, 
structured,  disciplined  development  techniques  and  methodologies,  and 
requirements  for  more  structured,  effective  documentation,  has  Increased 
the  feasibility  of  effective  measurement  of  software  quality. 

However,  before  the  potential  of  measurement  techniques  could  be  realized 
a  framework  or  model  of  software  quality  had  to  be  constructed.  An 


established  model,  which  at  one  level  provides  a  user  or  management 
oriented  view  of  quality,  is  described  In  Section  2  of  this  manual  in 
the  perspective  of  how  it  can  be  used  to  establish  software  quality 
requirements  for  a  specific  application. 

The  actual  measurement  of  software  quality  is  accomplished  by  applying 
software  metrics  (or  measurements)  to  the  documentation  and  source  code 
produced  during  a  software  development.  These  measurements  are  part  of 
the  established  model  of  software  quality  and  through  that  model  can 
be  related  to  various  user-oriented  aspects  of  software  quality. 

The  metrics  can  be  classified  according  to  three  categories: 

•  anomaly-detecting 

•  predictive 

•  acceptance 

Anomaly-detecting  metrics  identify  deficiencies  in  documentation  or  source 
code.  These  deficiencies  usually  are  corrected  to  Improve  the  quality  of 
the  software  product.  Standards  enforcement  is  a  form  of  anomaly-detecting 
metrics. 

Predictive  metrics  are  measurements  of  the  logic  of  the  design  and  imple¬ 
mentation.  These  measurements  are  concerned  with  form,  strucure,  density, 
complexity  type  attributes.  They  provide  an  Indication  of  the  quality  that 
will  be  achieved  in  the  end  product,  based  on  the  nature  of  the  application, 
and  design  and  implementation  strategies. 

Acceptance  metrics  are  measurements  that  are  applied  to  the  end  product 
to  assess  the  final  compliance  with  requirements.  Tests  are  a  form  of 
acceptance-type  measurements. 

The  measurements  described  and  used  in  this  manual  are  either  anomaly-detect¬ 
ing  or  predictive  metrics.  They  are  applied  during  the  development  phases 


3 


to  assist  In  identification  of  quality  problems  early  so  that  corrective 
actions  can  be  taken  early  when  they  are  more  effective  and  economical. 

The  measurement  concepts  complement  current  Quality  Assurance  and  testing 
practices.  They  are  not  a  replacement  for  any  current  techniques  utilized 
in  normal  quality  assurance  programs.  For  example,  a  major  objective  of 
quality  assurance  is  to  assure  conformance  with  user/customer  requirements. 
The  software  quality  metric  concepts  described  in  this  manual  provide  a 
methodology  for  the  user/customer  to  specify  life-cycle-oriented  quality 
requirements,  usually  not  considered,  and  a  mechanism  for  measuring  if  those 
requirements  have  been  attained.  A  function  usually  performed  by  quality 
assurance  personnel  is  a  review/audit  of  software  products  produced  during 
a  software  development.  The  software  metrics  add  formality  and  quantifica¬ 
tion  to  these  document  and  code  reviews.  The  metric  concepts  also  provide 
a  vehicle  for  early  involvement  in  the  development  since  there  are  metrics 
which  apply  to  the  documents  produced  early  in  the  development. 

Testing  is  usually  oriented  toward  correctness,  reliability,  and  perfor¬ 
mance  (efficiency).  The  metrics  assist  in  the  evaluation  of  other  qualities 
like  maintainability,  portability,  and  flexibility. 

A  summarization  of  how  the  software  metric  concepts  complement  Quality 
Assurance  activities  is  provided  in  Table  1.3-1  based  on  the  quality 
assurance  program  requirements  identified  in  MIL-S-52779. 


Table  1.3-1  How  Software  Metrics  Complement  Quality  Assurance 


QUALITY  ASSURANCE  IMPACT  OF  SOFTWARE  QUALITY 

PROGRAM  REQUIREMENTS  METRIC  CONCEPTS 


•  Assure  Conformance  with 
Requirements 


•  Identi fy  Software 
Deficiencies 

•  Provide  Configuration 
Management 

•  Conduct  Test 

•  Provide  Library  Controls 

•  Review  Computer  Program 
Design 

t  Assure  Software  Documentation 
Requirement  Conformance 

•  Conduct  Reviews  and  Audits 


t  Provide  Tools/Techniques/ 
Methodology  for  Quality 
Assurance 

•  Provide  Subcontractor  Control 


Adds  software  quality 
requi rements 

Anomaly-detecti ng 
metri cs 

No  impact 

Assists  in  evaluation  of 
other  qualities 

No  impact 

Predictive  metrics 

Metrics  assist  in  evaluation  of 
documentation  as  well  as  code 

Procedures  for  applying  metrics 
(in  form  of  worksheets)  formalizes 
inspection  process 

This  manual  describes  methodology 
of  using  metrics 

No  Impact 


All  of  these  concepts  will  be  further  explained  and  Illustrated  In  the  sub 
sequent  sections  of  this  manual. 


1.4  MANUAL  ORGANIZATION 

The  manual  has  been  organized  as  a  handbook  for  use  In  a  quality  assurance 
program.  The  first  section  provides  Introductory  Information  and  how  the 
manual  Is  to  be  used. 

The  second  section  defines  the  software  quality  model  and  describes  a 
methodology  for  using  this  model  to  establish  software  quality  requirements 
or  goals  for  a  software  development. 

The  third  section  describes  procedures  for  measuring  the  quality  of  the 
software.  These  procedures  cover  what  to  measure,  when  to  measure,  and 
how  to  measure. 

The  fourth  section  describes  procedures  for  utilizing  the  information 
provided  by  the  measurements  to  make  assessments  of  the  quality  of  the 
software  and  recommends  what  information  to  present  to  various  personnel 
involved  in  the  development. 

1.5  RECOMMENDED  USE  OF  MANUAL 

The  software  quality  metric  concepts  can  be  applied  at  several  levels. 

In  an  acquisition  manager/contractor  environment,  there  are  three  approaches 
for  using  the  metric  concepts.  They  are: 

1.  The  acquisition  manager's  staff  can  apply  metrics  to  the  delivered 
software  procucts. 

2.  The  development  manager's  staff  can  apply  metrics  to  software 
products  and  report  them  to  the  acquisition  manager  during 
reviews. 

3.  An  Independent  Quality  Assurance  contractor  can  apply  metrics  to 
delivered  software  products  and  report  them  to  the  acquisition 
manager. 


Within  the  software  development  project  organization,  there  are  two 
approaches  for  using  the  metric  concepts.  They  are: 

1.  The  quality  assurance  personnel  will  apply  the  metrics  as  an 
independent  assessment  of  the  quality  of  the  software  being  produced. 

2.  The  development  personnel  can  apply  the  metrics  during  walkthroughs 
and  reviews. 

This  manual  is  oriented  toward  those  personnel  who  will  be  applying  the 
concepts  (either  quality  assurance  or  development  personnel)  and  recommends 
three  approaches  to  both  establishing  the  quality  requirements  (Section  2) 
and  making  a  quality  assessment  (Section  4).  The  three  approaches  (an 
index  is  provided  in  Table  1.5-1)  in  each  area  are  presented  In  order  of 
increasing  formality  of  the  relationship  between  quality  requirements 
and  the  metrics,  that  is  in  order  of  increasing  quantification.  The  order 
of  presentation  also  relates  to  an  increasing  requirement  for  experience 
with  the  concepts  by  the  personnel  applying  the  concepts.  Thus,  the 
approaches  can  be  used  as  a  phased  implementation  plan  for  the  metric 
concepts.  It  is  recommended  that  the  concepts  be  incrementally  phased  into 
the  quality  assurance  organization's  operation. 

This  manual  should  be  utilized  by  the  personnel  applying  the  metric  con¬ 
cepts.  Additional  Information  and  definitions  can  be  found  In: 

"Factors  in  Software  Quality",  3  vols,  RADC-TR-77-369,  Nov  1977.|HcGA77] 

"Software  Quality  Metrics  Enhancements-Final  Report”,  Vol  I  of  this  document 

These  references  should  be  reau  oy  the  personnel  applying  the  metrics  to 
familiarize  them  with  the  underlying  concepts.  They  should  also  be  referred 
to  periodically  for  definitions  and  explanation  purposes, 


Table  1.5-1  Index  of  Three  Approaches  to  Specifying  and  Assessing 
Software  Quality 


APPROACH 
(LEVEL  OF 
FORMALITY) 

SPECIFYING 

SOFTWARE  QUALITY  . 

1 

Procedures  for 
Identifying  Important 
Quality  factors 
(Paragraph  2.2) 

2 

Procedures  for 
Identifying  critical 
software  attributes 
(Paragraph  2.3) 

3 

Procedures  for 

Identifying  quantifiable 
goals 

(Paragraph  2.4) 

APPLYING 

MEASUREMENTS 


PROCEDURES 

FOR 

APPLY IN6 
THE 
METRIC 
WORKSHEETS 
(SECTION  3) 


ASSESSING  THE  QUALITY  OF 
THE  PRODUCT 


Procedures  for  the 
Inspector's  assessment 
(Paragraph  4.2) 


Procedures  for  performing 
sensitivity  analysis 
(Paragraph  4.3) 


Procedures  for  use  of 
normalization  function 
(Paragraph  4.4) 


SECTION  2 
PROCEDURES  FOR 

INDENTIFYING  SOFTWARE  QUALITY  REQUIREMENTS 


2.1  INTRODUCTION 

The  primary  purpose  of  applying  Software  Quality  Metrics  In  a  Quality 
Assurance  Program  Is  to  Improve  the  quality  of  the  software  product. 

Rather  than  simply  measuring,  the  concepts  are  based  on  achieving  a 
positive  Influence  on  the  product,  to  Improve  Its  development. 

This  section  addresses  the  problem  of  Identifying  software  quality  require¬ 
ments  or  goals.  These  requirements  are  In  addition  to  the  functional, 
performance,  cost,  and  schedule  requirements  normally  specified  for  a 
software  development.  The  fact  that  the  goals  established  are  related  to 
the  quality  of  the  end  product  should.  In  Itself,  provide  some  positive 
Influence.  Past  research  has  shown  that  goal -directed  system  development 
Is  effective.  [WEIN72] 

The  vehicle  for  establishing  the  requirements  Is  the  hierarchical  model  of 
software  quality  defined  In  [CAVA78].  This  model,  shown  In  Figure  2.1-1, 
has  at  Its  highest  level  a  set  of  software  quality  factors  which  are 
user/ management -oriented  terms  and  represent  the  characteristics  which 
comprise  software  quality.  At  the  next  level  for  each  quality  factor.  Is 
a  set  of  criteria  which  are  the  attributes  If  present,  that  provide  the 
characteristics  represented  by  the  quality  factors.  The  criteria,  then, 
are  software- related  terms.  At  the  lowest  level  of  the  model  are  the 
metrics  which  are  quantitative  measures  of  the  software  attributes  defined 
by  the  criteria. 

The  procedures  for  establishing  the  quality  requirements  for  a  particular 
software  system  utilize  this  model  and  will  be  described  as  a  three  level 
approach,  the  levels  corresponding  to  the  hierachical  levels  of  the  software 


quality  model.  The  first  level  establishes  the  quality  factor?  that 
are  Important.  The  second  level  Identifies  the  critical  software 
attributes.  The  third  level  Identifies  the  metrics  which  will  be  applied 
and  establishes  Quantitative  ratings  for  the  quality  factors. 

Once  the  quality  requirements  have  been  determined  by  following  the 
procedures  described  In  the  subsequent  paragraphs,  they  must  be  trans- 
mltted  to  the  development  team.  In  a  formal  acquisition  manager/con¬ 
tractor  environment,  the  Request  for  Proposal  (RFP)  Is  the  medium  for 
Identifying  these  requirements.  The  results  of  following  the  procedures 
should  be  Incorporated  In  the  RFP.  If  the  development  is  being  done 
Internally,  the  quality  requirements  should  be  documented  In  the  same 
form  as  the  other  system  requirements  and  provided  to  the  development 
team.  Additionally,  a  briefing  emphasizing  the  Intent  of  the  Inclusion  of 
the  quality  requirements  Is  recommended. 


FRAMEWORK 


-  MANAGEMENT-ORIENTED 
VIEW  OF  PRODUCT  QUALITY 


SOFTWARE-ORIENTED 
ATTRIBUTES  WHICH 
PROVIDE  QUALITY 


-  QUANTITATIVE  MEASURES 
OF  THOSE  ATTRIBUTES 


Figure  2.1-1 


2.2  PROCEDURES  FOR  IDENTIFY IKS  IMPORTANT  QUALITY  FACTORS 


2.2.1  PROCEDURES 

The  basic  tool  to  be  utilized  in  identifying  the  Important  quality  factors 
will  be  the  Software  Quality  Requirements  Survey  form  shown  in  Table  2.2-1. 
The  formal  definitions  of  each  of  the  eleven  quality  factors  are  provided 
on  that  form. 

It  is  recommended  that  a  briefing  be  provided  to  the  decision  makers  using 
the  tables  and  figures  which  follow  in  this  paragraph  to  solicit  their 
responses  to  the  survey.  The  decision  makers  may  Include  the  acquisition 
manager,  the  user/customer,  the  development  manager,  and  the  QA  manager. 

To  complete  the  survey  the  following  procedures  should  be  followed: 

la.  Con&ideA  Bcuic  ChcuacteAiiticA  oi  the  Application 

The  software  quality  requirements  for  each  system  are  unique 
and  are  Influenced  by  system  or  application-dependent  character¬ 
istics.  There  are  basic  characteristics  which  affect  the 
quality  requirements  and  each  software  system  must  be  evaluated 

for  its  basic  characteristics.  Table  2.2-2  provides  a  list  of 
some  of  these  basic  characteristics.  For  example,  if  the  system 
is  being  developed  in  an  environment  in  which  there  is  a  high 
rate  of  technical  breakthroughs  in  hardware  design,  Portability 
should  take  on  an  added  significance.  If  the  expected  life  cycle 
of  the  system  is  long.  Maintainability  becomes  a  cost-critical 
consideration.  If  the  application  is  an  experimental  system 
where  the  software  specifications  will  have  a  high  rate  of 
change,  Flexibility  in  the  software  product  is  highly  desirable. 

If  the  functions  of  the  system  are  expected  to  be  required  for  a 
long  time,  while  the  system  itself  may  change  considerably. 
Resuability  Is  of  prime  Importance  in  those  modules  which 
Implement  the  major  functions  of  the  system.  With  the  advent  of 
more  computer  networks  and  communication  capabilities,  more 
systems  are  being  required  to  Interface  with  other  systems 


Table  2.2-1  Software  Quality  Requirements  Survey  Form 


1.  The  11  quality  factors  listed  below  have  been  Isolated  from  the  cur¬ 
rent  literature.  They  are  not  meant  to  be  exhaustive,  but  to  reflect 
what  Is  currently  thought  to  be  Important.  Please  Indicate  whether 
you  consider  each  factor  to  be  Very  Important  (VI),  Important  (I), 
Somewhat  Important  (SI),  or  Not  Important  (NI)  as  design  goals  In  the 
system  you  are  currently  working  on. 


RESPONSE  FACTONS 
_  CORRECTNESS 

_  RELIABILITY 

_  EFFICIENCY 

_ _  INTEGRITY 

_  USABILITY 

_  MAINTAINABILITY 

_  TESTABILITY 

_  FLEXIBILITY 

_  PORTABILITY 

_  REUSABILITY 

_  INTEROPERABILITY 


DEFINITION 

Extant  to  which  a  program  satisfies  its 
spaelfl  cations  and  fulfills  tha  user's 
msslon  objectives. 

Extant  to  uhlcb  a  program  can  be  expected 
to  perform  Its  Intended  function  with 
required  precision. 

The  mount  of  computing  resources  and  code 
required  by  a  program  to  perforo  a  function. 

Extant  to  ublch  access  to  software  or  data 
by  unauthorised  persons  can  be  controlled. 

Effort  required  to  learn,  operate,  prepare 
Input,  and  Interpret  output  of  e  program. 

Effort  required  to  locate  end  fix  an  error 
In  an  operational  program. 

Effort  required  to  toot  a  prog rao  to  Insure 
It  performs  Its  intended  faction. 

Effort  required  to  Modify  an  operational 
prog  ran. 

Effort  required  to  tronsfor  a  prog  ran  from 
one  hardware  configuration  and/or  software 
systan  environment  to  another. 

Extant  to  uhleh  a  program  can  bo  uood  In  other 
applications  -  relstad  to  the  packaging  and 
scopa  of  the  factions  thst  programs  perform. 

Effort  required  to  diuplo  one  systan  with 
another. 


2.  What  type(s)  of  application  are  you  currently  Involved  in? 


3.  Are  you  currently  in: 

_  1 .  Development  phase 

_  2.  Operations/Maintenance  phase 

4.  Please  Indicate  the  title  which  most  closely  describes  your  position: 

_ 1 .  Program  Manager 

_ 2.  Technical  Consultant 

_  3.  Systems  Analyst 

_ 4.  Other  (please  specify) _ _ 
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and  the  concept  of  interoperability  Is  extremely  Important,  These 
and  other  system  characteristics  should  be  considered  when  Identifying 
the  Important  quality  factors. 

Table  2.2-2  System  Characteristics  and  Related  Quality  Factors 


|  CHARACTERISTIC 

QUALITY  FACTOR 

t 

If  human  lives  are  affected 

Reliability 

Correctness 

Testability 

e 

Long  life  cycle 

Maintainability 

Flexibility 

Portability 

e 

Real  time  application 

Efficiency 

Reliability 

Correctness 

e 

On-board  computer 
application 

Efficiency 

Reliability 

Correctness 

e 

Processes  classified 

Information 

Integrity 

e 

Interrelated  systems 

Interoperability 

1b.  Con&ideA.  Lilt  Cycle  Implication* 

The  eleven  quality  factors  Identified  on  the  survey  can  be 
grouped  according  to  three  life  cycle  activities  associated 
with  a  delivered  software  product.  These  three  activities  are 
product  operation,  product  revision,  and  product  transition. 
The  relationship  of  the  quality  factors  to  these  activities 
is  shown  in  Table  2.2-3.  This  table  also  illustrates  where 
quality  indications  can  be  achieved  through  measurement  and 
where  the  impact  is  felt  if  poor  quality  is  realized.  The 
size  of  this  impact  determines  the  cost  savings  that  can  be 
expected  if  a  higher  quality  system  is  achieved  through  the 
application  of  the  metrics.  This  cost  savings  is  somewhat 
offset  by  the  cost  to  apply  the  metrics  and  the  cost  to 
develop  the  higher  quality  software  product  as  illustrated 
in  Figure  2.2-1. 


Figure  2.2-1 

Cost  vs  Benefit  Tradeoff 

This  cost  to  implement  versus  life  cycle  cost  reduction  relation¬ 
ship  exists  for  each  quality  factor.  The  benefit  versus  cost-to- 
provide  ratio  for  each  factor  is  rated  as  high,  medium,  or  low  in 
the  right  hand  column  of  Table  2.2-3.  This  relationship  and  the 
life  cycle  implications  of  the  quality  factors  should  be  considered 
when  selecting  the  important  factors  for  a  specific  system. 


LEGEND:  -  where  quality  factors  should  be  measured  X  -  where  impact  of  poor  quality  is  realized 


1c.  PeAfanm  TJiadeo^fA  Among  the  Tentative.  List  oi  Quality  Facto  fu>. 

As  a  result  of  steps  la  and  1b,  a  tentative  list  of  quality 
factors  should  be  produced.  The  next  step  is  to  consider  the 
interrelationships  among  the  factors  selected.  Table  2,2-4  and 
2.2-5  can  be  used  as  a  quide  for  determing  the  relationships 
between  the  quality  factors.  Some  factors  are  synergistic  while 
others  conflict.  The  impact  of  conflicting  factors  is  that 
the  cost  to  implement  will  increase.  This  will  lower  the 
benefit  to  cost  ratio  described  in  the  proceeding  paragraph. 


Table  2.2-4  Relationships  Between  Software  Quality  Factors 


LEGEHO 

If  a  high  degree  of  quality  Is  present  for  factor, 
what  degree  of  quality  Is  expected  for  the  other: 

0  ■  High  0  ■  low 

Blank  •  No  relationship  or  application  dependent 
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J* 

Table  2.2-5  Typical  Factor  Tradeoffs 


INTEGRITY 

VS 

EFFICIENCY 

The  additional  code  and  processing  required  to 
control  the  access  of  the  software  or  data 
usually  lengthens  run  time  and  require  additional 
storage. 

USABILITY 

The  additional  code  and  processing  required  to  ease 

VS 

an  operator's  tasks  or  provide  more  usable  output 

EFFICIENCY 

usually  lenghten  run  time  and  require  additional 
storage. 

MAINTAINABILITY 

Optimized  <code,  incorporating  intricate  coding 

VS 

techniques  and  direct  code,  always  provffees 

EFFICIENCY 

problems  to  the  maintainer.  Using  modularity, 
instrumentation, and  well  commented  high  level  code  to 
increase  the  maintainability  of  a  system  usually 
increases  the  overhead  resulting  in  less  efficient 
operation. 

TESTABILITY 

VS 

EFFICIENCY 

The  above  discussion  applies  to  testing. 

PORTABILITY 

The  use  of  direct  code  or  optimized  system  software 

VS 

or  utilities  decreases  the  portability  of  the 

EFFICIENCY 

system. 

FLEXIBILITY 

The  generality  required  for  a  flexible  system 

VS 

increases  overhead  and  decreases  the  efficiency 

EFFICIENCY 

of  the  system. 

REUSABILITY 

The  above  discussion  applies  to  reusability. 

VS 

\ 

EFFICIENCY 

INTEROPERABILITY 

AgVin  the  added  overhead  for  conversion  from 

VS 

standard  data  representations,  and  the  use  of 

EFFICIENCY 

interface  routines  decreases  the  operating 
efficiency  of  the  system. 

FLEXIBILITY 

Flexibility  requires  very  general  and  flexible 

VS 

data  structures.  This  increases  the  data  security 

INTEGRITY 

problem. 

REUSABILITY 

As  in  the  above  discussion,  the  generality  required 

VS 

by  reusable  software  provides  severe  protection 

INTEGRITY 

problems. 

INTEROPERABILITY 

Coupled  systems  allow  for  more  avenues  of  access  and 

VS 

different  users  who  can  access  the  system.  The 

INTEGRITY 

potential  for  accidental  access  of  sensitive  data  is 
increased  as  well  as  the  opportunities  for  deliberate 
access.  Often,  coupled  systems  share  data  or  soft¬ 
ware  which  compounds  the  security  problems  as  well. 

REUSABILITY 

The  generality  required  by  reusable  software  makes 

VS 

providing  error  tolerance  and  accuracy  for  all 

RELIABILITY 

cases  more  difficult. 
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ld.  Identify  M ot>t  Important  Quality  Factor 

Based  on  la  through  1c,  a  list  of  quality  factors  considered  to 
be  important  for  the  particular  system  should  be  compiled.  The 
list  should  be  organized  in  order  of  importance.  A  single 
decision  maker  may  choose  the  factors  or  the  choice  may  be  made 
by  averaging  several  survey  responses.  The  definitions  of  the 
factors  chosen  should  be  included  with  this  list. 

le.  Pnovide  Explanation  ion  Choice. 

The  rationale  for  the  decisions  made  during  steps  la  through  lc 
should  be  documented. 

2.2.2  AN  EXAMPLE  OF  FACTORS  SPECIFICATION 

To  illustrate  the  application  of  these  steps,  consider  a  tactical  in¬ 
ventory  control  system.  The  inventory  control  system  maintains  inven¬ 
tory  status  and  facilitates  requisitioning,  reordering,  and  issuing 
of  supplies  to  Amy  units  in  combat  situations.  The  planned  life 
of  the  system  is  ten  years. 

Each  step  described  previously  will  be  performed  with  respect  to  the 
tactical  inventory  control  system. 

la.  Con&ide a  Ba&ic  ChaAOJcWuAlicA  the  Application 

Utilizing  table  2.2-2  and  considering  the  unique  char¬ 
acteristics  of  the  tactical  inventory  control  system 
resulted  in  the  following: 

Characteristic  Related  Quality  Factor 

Critical  Support  ion  Reliability 

Combat  Unit*  Connectne** 

Long  Liie  Cycle  Maintainability 

With  Stable  Handoane 
And  So  itwane  Requitvement* 

Utilized  By  Amy  Supply  Usability 

Peuonnel 


lnten.ha.ee&  with  Inventory 
Sy&ten 16  At  Otken.  Atony 
Echelom 


InWwpeAabiZity 


lb.  Conbiden.  Life  CycJLe  ImptLcaXioni 


Of  the  five  quality  factors  identified  in  la,  all  provide 
high  or  medium  life  cycle  cost  benefits  according  to 
Table  2.2-3. 


FACTORS 

R letiabtity 

CoMuucJyiu* 

MalntaimbUJMj 

U&abiZLty 

lnteAopenobitiAy 


COST  BENEFIT  RATIO 

High 

High 

High 

Medium 

High 


lc.  PeA^oton  Inode.  0 Among  FacXou 

Using  Table  2.2-4,  there  are  no  conflicts  which  need  to  be 
considered. 


Id.  Identify  M o&t  Important  QuaLity  F actoto> 

Using  the  survey  form,  Table  2.2-1,  and  the  guidance 
provided  by  steps  la  through  lc,  the  following  factors 
are  identified  in  order  of  importance.  The  definitions 
are  provided. 

CORRECTNESS  -Extent  to  which  a  program  satisfies  its 

specifications  and  fulfills  the  user's 
mission  objectives. 

RELIABILITY  -Extent  to  which  a  program  can  be  expected 

to  perform  its  intended  function  with 
required  precision. 

USABILITY  -Effort  required  to  learn,  operate,  pre¬ 

pare  input,  and  interpret  output  of  a 
program. 


MAINTAINABILITY  -Effort  required  to  locate  and  fix  an 

error  In  an  operational  program. 

INTEROPERABILITY  -Effort  required  to  couple  one  system 

with  another. 

le.  Provide  Explanation  fan  Choice. 

CORRECTNESS  -System  performs  critical  supply  function 

RELIABILITY  -System  performs  critical  supply  functions 

In  field  environment 

USABILITY  -System  will  be  used  by  military  personnel 

with  minimum  computer  training 

MAINTAINABILITY  -System  life  cycle  is  projected  to  be  10 

years  and  will  operate  In  the  field  where 
military  personnel  will  maintain. 

INTEROPERABILITY  -System  will  Interface  with  other  supply 

systems  at  higher  levels  of  command 

2.3  PROCEDURES  FOR  IDENTIFYING  CRITICAL  SOFTWARE  ATTRIBUTES 
2.3.1  PROCEDURES 

The  next  level  of  Identifying  the  quality  requirements  Involves  proceeding 
from  the  user-oriented  quality  factors  to  the  software-oriented  criteria. 
Sets  of  criteria,  which  are  attributes  of  the  software,  are  related  to  the 
various  factors  by  definition.  Their  Identification  is  automatic  and 
represents  a  more  detailed  specification  of  the  quality  requirements. 

2a.  Identify  Cniticat  So^toane  Atttibutei  Requited 

Table  2.3-1  should  be  used  to  Identify  the  software  attributes 

associated  with  the  chosen  critical  quality  factors. 


Table  2.3-1  Software  Criteria  and  Related  Quality  Factors 


FACTOR 

SOFTWARE  CRITERIA 

FLEXIBILITY 

NODULARITY 

GENERALITY 

EXPANDABILITY 

TESTABILITY 

SIMPLICITY 

MODULARITY 

INSTRUMENTATION 

SELF-DESCRIPTIVENESS 

PORTABILITY 

MODULARITY 

SELF-DESCRIPTIVENESS 
MACHINE  INDEPENDENCE 
SOFTWARE  SYSTEM 

INDEPENDENCE 

REUSABILITY 

GENERALITY 

MODULARITY 

SOFTWARE  SYSTEM 

INDEPENDENCE 

INTEROPERABILITY 

MODULARITY 

COMMUNICATION 

COMMONALITY 
DATA  COMMONALITY 

FACTOR 

SOFTWARE  CRITERIA 

CORRECTNESS 

TRACEABILITY 

CONSISTENCY 

COWLETENESS 

RELIABILITY 

ERROR  TOLERANCE 
CONSISTENCY 

ACCURACY 

SIMPLICITY 

EFFICIENCY 

STORAGE  EFFICIENCY 
EXECUTION  EFFICIENCY 

INTEGRITY 

ACCESS  CONTROL 

ACCESS  AUDIT 

USABILITY 

OPERABILITY 

TRAINING 

COMMUNICATIVENESS 

MAINTAINABILITY 

CONSISTENCY 

2b.  Ptouufe.  Ve^uLctiom 

The  definitions  In  Table  2.3-2  should  also  be  provided  as  part 
of  the  specification. 


Table  2.3-2  Criteria  Definitions  for  Software  Quality 


Factors 


CRITERION 

DEFINITION 

TRACEABILITY 

Those  attrl butts  of  the  software  that  provide 
a  thrtad  from  th*  requirements  to  the  imple¬ 
mentation  with  ratpact  to  tha  (pacific 
development  and  operational  environment. 

COMPLETENESS 

Those  attributes  of  the  software  that 
provide  full  Implementation  of  the  functions 
required. 

CONSISTENCY 

Those  attributes  of  the  software  that 
provide  uni  fora  design  and  lap I amen tat Ion 
techniques  and  notation. 

ACCURACY 

Those  attributes  of  tha  software  that 
provide  tha  required  precision  In  calcula¬ 
tions  and  outputs. 

ERROR  TOLERANCE 

Those  attributes  of  the  software  that 
provide  continuity  of  operation  under 
nonnoalnal  conditions. 

SIMPLICITY 

Those  attributes  of  tha  software  that 
provide  inpleaentatlon  of  functions  In  tha 
most  understandable  manner.  (Usually 
avoidance  of  practices  which  Increese 
complexity.) 

MODULARITY 

Those  attributes  of  the  software  that 
provide  a  structure  of  highly  independent 
nodules. 

GENERALITY 

Those  attributes  of  the  software  that 
provide  breadth  to  the  functions  perfonied. 

EXPANDABILITY 

Those  attributes  of  the  software  that 
provide  for  expansion  of  data  storage 
requirements  or  computational  functions. 

INSTRUtCNTATION 

Those  attributes  of  the  software  that 
provide  for  the  measurement  of  usage  or 
Identification  of  errors. 

SELf- 

DESCRIPTIVENESS 

Those  attributes  of  the  software  that 
provide  explanation  of  the  Implementation 
of  a  function. 

Table  2.3-2  Criteria 


DATA  COAWHALm 


Definitions  for  Software  Quality  Factors 
(Continued) 

— 

DEFINITION  i 


Those  attributes  of  tne  software  tut 
provide  for  minimum  processing  tine. 

Those  attributes  of  tne  software  thet 
provide  for  minimum  storege  requirements  , 
during  operation. 

■  1  -I  I  .1  -I.  — -  II.  .1  - .  Ml.  j 

Those  tttrlbutes  of  the  software  that  i 
provide  for  control  of  the  access  of 
software  and  data.  i 

Those  attributes  of  the  software  that  > 

provide  for  an  audit  of  the  access  of  I 

software  and  data. 

J - 1 

Those  attributes  of  the  software  that 
determine  operation  and  procedures  con* 
earned  with  the  operation  of  the  software.; 

Those  attributes  of  the  software  that 
provide  transition  from  current  operation  ; 
or  initial  famillariiation.  , 

Those  attributes  of  the  soft**r»  that 
provide  useful  Inputs  and  outputs  wnlch 
can  be  assimilated.  i 

Those  attributes  of  the  software  tut  ; 

determine  its  dependency  on  the  software  i 
environment  (operating  systems,  utilities, 
Input/output  routines,  etc.)  j 

Those  ittributts  of  the  software  that  1 

determine  its  dependency  on  tu  hardware 
system. 

Those  attributes  of  the  software  that 
provide  the  use  of  standard  protocols 
and  interface  routines. 

Those  attributes  of  the  software  that 
provide  the  use  of  standard  data  repre¬ 
sentations  .  i 

Those  attributes  of  the  software  that 
provide  for  implementation  of  a  function  j 
with  a  minimum  amount  of  code. 
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2.3.2  EXAMPLE  OF  IDENTIFYING  SOFTWARE  CRITERIA 

Continuing  with  the  example  of  paragraph  2.2.2,  the  software  criteria 

for  the  Identified  quality  factors  would  be  chosen. 


2a.  Jdzntiiy  OiiticAt  So&tua/ te.  Attributes 

Using  the  relationships  provided  In  Table  2.3-1,  the  following 
criteria  would  be  Identified. 


CRITERIA 


RELATED  FACTOR 


TRACEABILITY. 

CONSISTENCY. 

COMPLETENESS 
ERROR  TOLERANCE. 

ACCURACY. 

SIMPLICITY: 

CONCISENESS 
MODULARITY 
SELF-PESCRIPTIVENESS 
OPERABILITY. 

TRAINING 

COMMUNICATIVENESS 
COMMUNICATIONS  COMMONALITY- 
DATA  COMMONALITY. 


CORRECTNESS 

RELIABILITY 

MAINTAINABILITY 

USABILITY 

INTEROPERABILITY 


2b.  Pttov4.de.  VefinCUoni 

The  definitions  fbr  each  of  these  criteria  would  be  provided 
also. 


2.4.1  PROCEDURES 

The  last  level,  which  Is  the  most  detailed  and  quantified,  requires  precise 
statements  of  the  level  of  quality  that  will  be  acceptable  for  the  software 
product. 


Currently,  the  underlying  mathematical  relationships  which  allow  measurement 
at  this  level  of  precision  do  not  exist  for  all  of  the  quality  factors.  The 
mechanism  for  making  the  precise  statement  for  any  quality  factor  is  a  rating 
of  the  factor.  The  underlying  basis  for  the  ratings  Is  the  effort  or  cost 
required  to  perform  a  function  such  as  to  correct  or  modify  the  design  or 
program.  For  example,  rating  for  maintainability  might  be  that  the  average 
time  to  fix  a  problem  should  be  five  man-days  or  that  90%  of  the  problem 
fixes  should  take  less  than  six  man-days.  This  rating  would  be  specified 
as  a  quality  requirement.  To  comply  with  this  specification,  the  software 
would  have  to  exhibit  characteristics  which,  when  present,  given  an  indication 
that  the  software  will  perform  to  this  rating.  These  characteristics  are 
measured  by  metrics  which  are  inserted  into  a  mathematical  relationship  to 
obtain  the  predicted  rating. 


In  order  to  choose  ratings  such  as  the  two  mentioned  above,  data  must  be 
available  which  allows  the  decision  maker  to  know  what  is  a  "good  rating"  or 
perhaps  what  Is  the  Industry  average.  Currently  there  is  generally  a  lack 
of  good  historical  data  to  establish  these  expected  levels  of  operations  and 
maintenance  performance  for  software.  There  are  significant  efforts  under¬ 
way  to  compile  historical  data  and  derive  the  associated  performance  statis¬ 
tics  [DUVA76].  Individual  software  development  organizations  and  System 
Program  Offices  should  attempt  to  compile  historical  data  for  their  particular 
environment.  Any  environment-unique  data  available  should  be  used  as  a 
check  against  the  data  provided  as  guidelines  In  this  manual.  The  data 
utilized  in  this  section  Is  based  on  experiences  applying  the  metrics  to 
several  large  comnand  and  control  software  systems  and  other  experiences 
reported  in  the  literature. 
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3a.  Speech  Rating  fan.  Each  QualAtiy  faction. 

After  identification  of  the  critical  quality  factors,  specific  perform¬ 
ance  levels  or  ratings  required  for  each  factor  should  be  specified. 

Table  2.4.1  should  be  used  as  a  guideline  for  Identifying  the  ratings  * 
for  the  particular  factors.  Note  that  mathematical  relationships  have 
not  been  established  for  some  of  the  factors.  In  those  cases.  It  is 
advisable  not  to  levy  requirements  for  meeting  a  specific  quality 
rating  but  instead  specify  the  relative  Importance  of  the  quality 
factor  as  a  development  goal.  Not  that  the  reliability  ratings  are 
provided  In  terms  familiar  to  traditional  hardware  reliability.  Just 
as  in  hardware  reliability  there  are  significant  differences  between 
ratings  of  .9  and  .99. 

3b.  Identify  Spcctifac.  UetinAc*  to  be.  A ppLLed 

The  next  step  or  an  alternative  to  3a  Is  to  Identify  the  specific  metrics 
.  which  will  be  applied  to  the  various  software  products  produced  during 
the  development.  The  Metric  Worksheets  described  ip  Section  3  can  be 
used  for  this  purpose  or  Table  2.4-2  can  be  used  to  identify  the  metrics 
and  reference  can  be  made  to  RADC-TR-79-  [MCCA79]  where  definitions  of 
the  metrics  are  provided. 

\ 

3c.  SpccAfajcatiLon  o£  MetinAjc  Thne*hold  Value* 

In  lieu  of  specifying  quality  ratings  or  In  addition  to  the  ratings, 
specific  minimum  values  for  particular  metrics  may  be  specified.  This 
technique  is  equivalent  to  establishing  a  standard  which  Is  to  be  adhered 
to.  Violations  to  the  value  established  are  to  be  reported.  Typical 
values  can  be  derived  by  applying  the  metrics  to  software  products 
developed  in  a  particular  environment  or  by  looking  at  the  scores  re¬ 
ported  In  [MCCA77]  and  [MCCA79].  When  establishing  these  threshold 
values  based  on  past  project  data,  projects  which  have  been  considered 
sucessful,  l.e.,  have  demonstrated  good  characteristics  during  their 
life  cycle  should  be  chosen.  For  example,  a  system  which  has  been 
relatively  cost-effective  to  maintain  over  Its  operational  history 
should  be  chosen  to  apply  the  metrics  related  to  maintainability  and 
establish  threshold  values. 


Table  2.4-1  Quality  Factor  Ratings 


QUALITY  FACTOR 

RATING  EXPLANATION 

RATING  GUIDELINES 

RELIABILITY  * 

Rating  is  in  terms  of  the 
number  of  errors  that  occur 
after  the  start  of  formal 
testing. 

Rati  no  =  1-  Nuntoer  of  Errors 
Nunber  of  Lines  of 
source  code  exclud¬ 
ing  comments 

RATING 

.9 

.98** 

.99 

.999 

ERRORS 

10 

2 

1 

.1 

160  Loc 

MAINTAINABILITY  * 

Rating  is  in  terms  of  the 
average  amount  of  effort 
required  to  locate  and  fix 
an  error  in  an  operational 
program. 

Rating  =  l-.l  (Average  number 
of  man  days 
per  fix) 

RATING 

.3 

.5 

.7** 

.9 

AVERAGE 

EFFORT 

(MAN 

DAYS) 

7 

5 

3 

1 

PORTABILITY  * 

Rating  Is  In  terms  of  the 
effort  required  to  convert  a 
program  to  run  in  another 
environment  with  respect 
to  the  effort  required  to 
orglnally  Implement  the 
program. 

Ratine  *  1-  Effort  to  Transport 

RATING 

.25 

.5** 

.75 

.9 

%  OF 

ORIGINAL 

EFFORT 

75 

50 

25 

10 

Effort  to  Implement 

FLEXIBILITY  * 

Rating  Is  In  terms  of  the 
average  effort  required  to 
extend  a  program  to  Include 
other  requirements. 

Rating  *  1-. 05  (Average  number 
of  man  days  to 
change ) 

RATING 

.3 

.5  ** 

.7 

.9 

AVERAGE 

EFFORT 

(MAN 

DAYS) 

14 

10 

6 

2 

(Continued) 
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Table  2.4-1  (Continued) 


QUALITY  FACTOR 

RATING  EXPLANATION 

CORRECTNESS 

The  function  which  the  software  is  to  perform  is 
incorrect.  The  rating  is  in  terms  of  effort 
required  to  Implement  the  correct  function. 

EFFICIENCY 

The  software  does  not  meet  performance  (speed, 
storage)  requirements.  The  rating  is  in  terms 
of  effort  required  to  modify  software  to  meet 
performance  requirements. 

INTEGRITY 

The  software  does  not  provide  required  security. 
The  rating  is  in  terms  of  effort  required  to 
implement  proper  levels  of  security. 

USABILITY 

There  is  a  problem  related  to  operation  of  the 
software,  the  user  interface,  or  the  input/ 
output.  The  rating  is  in  terms  of  effort 
required  to  improve  human  factors  to  acceptable 
level. 

TESTABILITY 

The  rating  is  in  terms  of  effort  requried  to  test 
changes  or  fixes. 

REUSABILITY 

The  rating  Is  in  terms  of  effort  required  to  use 
software  In  a  different  application. 

INTEROPERABILITY 

The  rating  is  in  terms  of  effort  required  to  coupl 
the  system  to  another  system. 

NOTES 

*  Data  collected  to  date  provides  some  basis  upon  which  to  allow 
quantitative  ratings  for  these  quality  factors.  These  ratings 
should  be  modified  based  on  data  collected  within  a  specific 
development  environment.  Data  has  not  been  collected  to  support 
ratings  of  the  other  quality  factors. 

**  Indicates  rating  which  might  be  considered  current  industry  average. 


Table  2.4-2  Quality  Metrics  Related  to  Factors 


FACTOR 


CORRECTNESS 


RELIABILITY 


EFFICIENCY 


METRICS  * 


TRACEABILITY  CHECK  (TR.l) 
COMPLETENESS  CHECKLIST  (CP.l) 
CONSISTENCY  CHECKLISTS  (CS.1-2) 


ACCURACY  CHECKLIST  (AY.l) 

ERROR  TOLERANCE  CHECKLISTS  (ET.l-S) 
DESIGN  STRUCTURE  MEASURE  (ST.l) 
STRUCTURED  PROGRAMING  CHECK  (SI. 2) 
COMPLEXITY  MEASURE  (SI. 3) 

CODING  SIMPLICITY  MEASURE  (SI. 4) 


PERFORMANCE  REQUIREMENTS  CHECK  (EE.l) 
ITERATIVE  PROCESSING  MEASURE  (EE. 2) 
DATA  USAGE  MEASURE  (EE. 3) 

STORAGE  EFFICIENCY  MEASURE  (EE. 4) 


INTEGRITY 


ACCESS  CONTROL  CHECKLIST  (AC.l 
ACCESS  AUDIT  CHECKLIST  (AA.l) 


) 


USABILITY  OPERABILITY  CHECKLIST  (OP.l) 

TRAINING  CHECKLIST  (TR.l) 

USER  INPUT  INTERFACE  MEASURE  (CM.l) 
_ _  USER  OUTPUT  INTERFACE  MEASURE  (CM. 2) 


MAINTAINABILITY  CONSISTENCY  CHECKLISTS  (CS1-2) 

DESIGN  STRUCTURE  MEASURE  (SI .1 ) 

STRUCTURE  LANGUAGE  CHECK  (SI. 2) 

COMPLEXITY  MEASURE  (SI. 3 
CODING  SIMPLICITY  MEASURE  (SI. 4) 

STABILITY  MEASURE  (MO.l) 

MODULAR  INPLEMENTATION  MEASURE  (MO. 2) 
QUANTITY  OF  COMMENTS  (SD.l) 

EFFECTIVENESS  OF  COMMENTS  MEASURE  (SD.2) 
DESCRIPTIVENESS  OF  IMPLEMENTATION  LANGUAGE 
MEASURE  (SD.3) 
CONCISENESS  MEASURE  (CO. 1) 


*  Acronym  references  in  parentheses  relate  to  definitions  in  [  MCCA79  ]. 

(Continued) 


Table  2.4-2  Quality  Metrics  Related  to  Factors  (Continued) 


FACTOR 


FLEXIBILITY 


TESTABILITY 


PORTABILITY 


REUSEABILITY 


METRICS  * 


MODULAR  IMPLEMENTATION  MEASURE  (MO. 2) 
INTERFACE  MEASURE  (GE.l) 

GENERALITY  CHECKLIST  (GE.2) 

DATA  STORAGE  EXPANSION  MEASURE  (EX.l) 
EXTENSIBILITY  MEASURE  (EX.l) 

QUANTITY  OF  COMMENTS  (SD.l) 

EFFECTIVENESS  OF  COMMENTS  MEASURE  (SD.2) 
DESCRIPTIVENESS  OF  IMPLEMENTATION  LANGUAGE 
MEASURE  (SD.3) 


DESIGN  STRUCTURE  MEASURE  (SI  .1 ) 

STRUCTURED  PROGRAMMING  CHECK  (SI. 2) 
COMPLEXITY  MEASURE  (SI. 3) 

CODING  SIMPLICITY  MEASURE  (SI. 4) 

STABILITY  MEASURE  (MO.  1) 

MODULAR  IMPLEMENTATION  MEASURE  (MO. 2) 

TEST  CHECKLISTS  (IN. 1-3) 

QUANTITY  OF  COMMENTS  (SD.l) 

EFFECTIVENESS  OF  COMMENTS  MEASURE  (SD.2) 
DESCRIPTIVENESS  OF  IMPLEMENTATION  LANGUAGE 
MEASURE  (SD.31 


MODULAR  IMPLEMENTATION  MEASURE  (MO. 2) 
QUANTITY  OF  COMMENTS  (SD.l) 

EFFECTIVENESS  OF  COMMENTS  MEASURE  (SD.2) 
DESCRIPTIVENESS  OF  IMPLEMENTATION  LANGUAGE 
MEASURE  (SD.3) 

SOFTWARE  SYSTEM  INDEPENDENCE  MEASURE  (SS.l) 
MACHINE  INDEPENDENCE  MEASURE  (MI.l) 


MODULAR  IMPLEMENTATION  MEASURE  (MO. 2) 
INTERFACE  MEASURE  (GE.l) 

GENERALITY  CHECKLIST  (GE.2) 

QUANTITY  OF  COMMENTS  (SD.l) 

EFFECTIVENESS  OF  COMMENTS  MEASURE  (SD.2) 
DESCRIPTIVENESS  OF  IMPLEMENTATION  LANGUAGE 
MEASURE  (SD.3) 

SOFTWARE  SYSTEM  INDEPENDENCE  MEASURE  (SS.l) 
MACHINE  INDEPENDENCE  MEASURE  (MI.l) 


INTEROPERABILITY 


MODULAR  IMPLEMENTATION  MEASURE  (MO. 2) 
COMMUNICATIONS  COMMONALITY  CHECKLIST  (CC.l) 
DATA  COMMONALITY  CHECKLIST  (DC.l) 


2.4.2  EXAMPLE  OF  METRICS 

Using  the  example  of  paragraph  2.2.2,  the  quality  ratings  would  be  specified 
as  follows. 

3a  Spectate  Quality  facto K  Rating* 

Ratings  for  two  of  the  five  important  quality  factors  can  be 


established  using 

Table  2.4-1. 

Reliability 

.99 

Require  less  than  one  error 
per  100  lines  of  code  to  be 
detected  during  formal  testing. 

Maintainability 

.8 

Require  2  man  days  as  an  average 
level  of  maintenance  for  correcting 

an  error. 

These  ratings  can 
the  development  as 

also  be  established  at  each  measurement  period  during 

follows: 

Reliability 

Maintainability 

REQ  POR 

TT  75~ 

.7  .7 

CDR  IMPL  ACCEPTANCE 

75“  "75“  - 755 — 

.8  .8  .8 

The  progressively  better  scores  are  required  because  there  Is 
more  detailed  information  in  the  later  phases  of  the  development 
to  which  to  apply  the  metrics  and  more  confidence  in  the  metrics' 
indication  of  quality.  This  Is  analagous  to  the  concept  of 
reliability  growth.  For  other  quality  factors  see  step  3b. 

3b  Identify  Specific  Metric*  to  be  Applied 

The  metrics  to  be  applied  to  assess  the  level  of  each  important 
quality  factor  are  chosen  from  Table  2.4-2.  A  subset  is  shown 
on  the  following  page: 
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Accuracy  Checklist - 

Error  Tolerance  Checklist. 


Complexity  Measure: 

Coding  Simplicity  Measure: 


Modular  Implementation  Measure 


Quantity  of  Comments - 

Effectiveness  of  Comments. 


RELIABILITY 


MAINTAINABILITY 


Traceability  Check _ _ 

Completeness  Checklist 
Consistency  Checklists 
Operability  Checklist. 


User  Interface  Checklists. 
Communications  Commonality- 
Data  Commonality _ 


CORRECTNESS 

USABILITY 

INTEROPERABILITY 


3c  Spe.cA.6y  Tf in.e*hold  Value* 

The  following  threshold  values  are  established  based  on  past 
experience  and  to  provide  a  goal  for  the  quality  factors  that 
were  not  given  ratings.  They  were  derived  by  determining  the 
average  scores  of  past  applications  of  the  metrics. 


Quantity  of  Comments  .2 

Effectiveness  of  Conments  .6 

Complexity  Measure  .1 

Traceability  Check  .9 

Completeness  Checklist  1. 

Consistency  Checklist  .9 

Operability  Checklist  .6 

Training  Checklist  .75 

User  Interface  Checklist  .75 

Communications  Commonality  .8 

Data  Conmonality  .8 
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2.5  EVALUATION  OF  DEVELOPMENT  PLAN 

In  an  acquisition  environment  the  initial  benefits  of  utilizing  the  quality 
metrics  concepts  are  realized  in  the  source  selection  process.  The  acquisi¬ 
tion  office  should  include  the  quality  goals  established  as  software  requirements 
in  the  Request  for  Proposal.  The  software  attributes  should  be  also  identi¬ 
fied  as  required  characteristics  in  the  software  and  the  metrics  as  the 
vehicles  for  assessing  their  existence.  The  bidders  should  be  required  to 
describe  how  they  plan  to  provide  those  characteristics  in  the  software. 

This  discussion  should  be  provided  in  the  portion  of  the  proposal  that 
describes  their  development  plan. 

The  description  of  the  bidder^  approach  for  including  the  required  attributes 
in  the  software  not  only  forces  acknowledgement  of  these  additional  require¬ 
ments  but  also  provides  additional  information  with  which  to  evaluate  the 
bidders  during  source  selection. 
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SECTION  3 

PROCEDURES  FOR  APPLYING  MEASUREMENTS 


3.1  WHEN  TO  APPLY  MEASUREMENTS 

The  software  quality  metrics  are  oriented  toward  the  availability  of 
information  about  the  software  system  as  it  progresses  in  its  development. 
In  the  early  phases  of  the  development,  the  metrics  are  applied  to  the 
documentation  produced  to  describe  the  concepts  of  the  system  and  its 
design.  In  the  later  phases  the  metrics  are  oriented  not  only  to  documen 
tation  but  also  to  the  source  code  that  is  available. 


Thus,  the  application  of  the  metrics  logically  follows  the  phased  devel¬ 
opment  of  software.  The  first  application  of  the  metric  is  at  the  end 
of  the  requirements  analysis.  The  next  application  is  during  design. 

If  the  design  phase  has  been  decomposed  into  a  preliminary  design  phase 
and  a  detailed  design  phase,  the  metrics  should  be  applied  at  the  end 
of  each  of  those  phases.  During  implementation,  i.e.  coding  the  metrics 
oriented  toward  the  source  code  should  be  applied  periodically  to  assess 
the  quality  growth  exhibited  as  the  code  evolves.  The  timing  of  the 
application  of  the  metrics  is  shown  in  Figure  3.1-1.  The  application  of 
the  metrics  can  be  done  during  or  just  proir  to  formal  customer  reviews 
(as  shown  in  Figure  3.1-1)  or  during  equivalent  activities  conducted  by 
the  development  personnel. 


REQUIREMENTS 

ANALYSIS 

DESIGN 

PROGRAMMING 

AND 

CHECKOUT 

TEST 

INTEGRATION 

TT 

n - n 

A 

REQUIREMENT 

REVIEW 


A 

PRELIMINARY 

DESIGN 

REVIEW 


AA 


A 

CRITICAL 

DESIGN 

REVIEW 


AAA.AA 

PERIODIC 
APPLICATION 
DURING 
COOING 
AND 

testing 


VALIDATION 

AND 

ACCEPTANCE 

TEST 

REVIEW 


A 

ACCEPTANCE 
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Figure  3.1-1  Timing  of  Metrics  Application 


3.2  SOURCES  OF  QUALITY  INFORMATION 

A  typical  minimum  set  of  documents  and  source  code  are  shown  in  Figure  3.2-1. 
These  documents  plus  the  source  code  are  the  sources  of  the  quality  infor¬ 
mation  derived  from  the  application  of  the  metrics. 


REQUIREMENTS 

ANALYSIS 

DESIGN 

PROGRAMMING 

AND 

CHECKOUT 

TEST 

AND 

INTEGRATION 

T 


A 

REQUIREMENTS 

SPEC 


A 


PRELIMINARY 

DESIGN 

SPEC 


USER'S  MANUAL 
(DRAFT) 


A 

DETAILED 

DESIGN 

SPEC 

TEST  PLAN 
AND 

PROCEDURES 


A 

SOURCE  CODE 

DETAILED 

DESIGN 

SPEC 

UPDATED 


A 

TEST 

RESULTS 


Figure  3.2-1 


USER'S  MANUAL 
(FINAL) 
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3.3  APPLICATION  OF  THE  MEASUREMENTS 

The  vehicle  for  applying  the  software  metrics  are  the  metric  worksheets 
contained  in  this  section  of  the  manual.  The  procedure  is  to  take  the 
available  documentation/source  code,  apply  the  appropriate  worksheet 
(as  shown  in  Figure  3.3-1),  and  translate  the  measurements  to  metric 
scores. 

The  worksheets  are  organized  as  follows.  In  the  header  portion  of  the  work¬ 
sheet  is  the  worksheet  identification  which  (1)  identifies  the  phase 
during  which  the  worksheet  is  initially  used  and  the  level  (system 
or  module)  to  which  the  worksheet  applies,  (2)  provides  identification  of 
the  system  and  the  module  to  which  the  worksheet  has  been  applied,  and  (3) 
provides  identification  of  the  date  and  the  inspector  who  took  the  measure¬ 
ments.  The  remaining  portion  of  each  worksheet  contains  the  measurements  to 
be  taken  and  questions  to  be  answered.  These  measurements  and  questions  are 
organized  by  quality  factors  identified  in  parentheses.  Each  logical 
group  of  measurements  and  questions  have  a  group  identifier  and  are 
bordered.  When  applying  the  measurements,  only  those  in  the  groups  that 
relate  to  the  quality  factors  chosen  as  quality  goals  should  be  applied. 

Metric  Worksheet  #  1  and  #  2a  contain  system  level  metrics  and  are  applied 
at  the  system  or  major  subsystem  (CPCI)  level  to  the  System  Requirements 
Specification,  the  Preliminary  Design  Specification,  the  User's  Manual,  and 
the  Test  documentation. 

Metric  Worksheets  #  2b  and  #3  contain  module  level  metrics  and  are  applied 
to  each  module's  design  (Detailed  Design  Specification)  and  implementation 
(source  code). 

L^efinitions  and  interpretations  of  the  individual  measurements  contained 
in  the  worksheets  are  found  in  Appendix  B  of  the  first  volume.  Next  to 
each  measurement  element  on  the  worksheets  is  an  index  into  the  metric 
table  in  Appendix  B. 


As  shown  In  Figure  3.3-1,  the  worksheets  may  be  applied  several  times  during 
the  development.  For  example.  Metric  Worksheet  #2b  which  Is  applied 
for  each  module  to  the  detailed  design  document  during  design  is  also 
applied  to  the  detailed  design  document  after  it  has  been  updated  to 
reflect  the  actual  implementation.  The  worksheet  does  not  have  to  be 
totally  reapplied.  The  successive  applications  of  any  worksheet  should 
require  considerably  less  effort  than  the  original  application.  The 
successive  applications  should  involve  simply  updates  to  reflect  the 
updates  made  to  the  system  since  the  previous  application  of  the  worksheet. 
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Figure  3.3-1  Application  of  the  Metric  Worksheets 


METRIC  WORKSHEET  I 

REQUIREMENTS  ANALYSIS/SYSTEM  LEVEL 


DATE _ 

INSPECTOR: 


I.  COMPLETENESS  (CORRECTNESS,  RELIABILITY) 


1.  Number  of  major  functions  identified  (equivalent  to  CPCI).  CP.l 

2.  Are  requirements  itemized  so  that  the  various  functions  to  be  performed,  their 
inputs  and  outputs,  are  clearly  delineated?  CP. 1(1) 

3.  Number  of  major  data  references.  CP.l (2) 

4.  How  many  of  these  data  references  are  not  defined?  CP. 1(2) 

5.  How  many  defined  functions  are  not  used?  CP. 1(3) 

6.  How  many  referenced  functions  are  not  defined?  CP.l (4) 

7.  How  many  data  references  are  not  used?  CP.l (2) 

8.  How  many  referenced  data  references  are  not  defined?  CP. 1(6) 

9.  Is  the  flow  of  processing  and  all  decision  points  in  that  flow  described?  CP. 1(5 

10.  How  many  problem  reports  related  to  the  requirements  have  been  recorded?  CP. 1(7) 

11.  How  many  of  those  problem  reports  have  been  closed  (resolved)?  CP.l (7) 


II.  PRECISION  (RELIABILITY) 


1.  Has  an  error  analysis  been  performed  and  budgeted  to  functions?  AY.l(l) 

2.  Are  there  definitive  statements  of  the  accuracy  requirements  for  inputs, 
outputs,  processing,  and  constants?  AY. 1(2) 

3.  Are  there  definitive  statements  of  the  error  tolerance  of  input  data?  ET.2(1) 

4.  Are  there  definitive  statements  of  the  requirements  for  recovery  from 
computational  failures?  ET. 3(1 ) 

5.  Is  there  a  definitive  statement  of  the  requirement  for  recovery  from  hardware 
faults?  ET.4(1) 

6.  Is  there  a  definitive  statement  of  the  requirements  for  recovery  from  device 
errors?  ET.5(1) 


III.  SECURITY  (INTEGRITY) 


1.  Is  there  a  definitive  statement  of  the  requirements  for  user  input/output 
access  controls?  AC .1(1) 

2.  Is  there  a  definitive  statement  of  the  requirements  for  data  base  access 
controls?  AC.  1(2) 

3.  Is  there  a  definitive  statement  of  the  requirements  for  memory  protection 
across  tasks?  AC. 1(3) 

4.  Is  there  a  definiti-t  statement  of  the  requirements  for  recording  and 
reporting  access  to  system?  AA.l(l) 

5.  Is  there  a  definitive  statement  of  the  requirements  for  immediate 
indication  of  iccess  violation?  AA.1(2) 


Pg.  2 


METRIC  WORKSHEET  1 

SYSTEM 

DATE 

REQUIREMENTS  ANALYSIS/SYSTEM  LEVEL 

NAME: 

INSPECTOR: 

IV.  HUMAN  INTERFACE  (USABILITY  ) 


1.  Are  all  steps  in  the  operation  described  (operations  concept)?  OP. 1(1) 

2.  Are  all  error  conditions  to  be  reported  to  operator/user  identified  and 
the  responses  described?  OP. 1(2) 

3.  Is  there  a  statement  of  the  requirement  for  the  capability  to  interrupt 
operation,  obtain  status,  modify,  and  continue  processing?  OP. 1(3) 

4.  Is  there  a  definitive  statement  of  requirements  for  optional  inputcmed|a’ 

5.  Is  there  a  definitive  statement  of  requirements  for  optional  output  media? 

CM.  2(7) 

6.  Is  there  a  definitive  statement  of  retirements  for  selective  output 
control?  CM  .2(1) 


V.  PERFORMANCE  (EFFICIENCY) 

1.  Have  performance  requirements  (storage  and  run  time)  been  identified  for 
the  functions  to  be  performed?  EE.l 

— ! — 

y  ;  n 

VI.  SYSTEM  INTERFACES  (INTEROPERABILITY)  | 

1.  Is  there  a  definitive  statement  of  the  requirements  for  communication  with 
other  systems?  CC.l(l) 

■ 

N 

2.  Is  there  a  definitive  statement  of  the  requirements  for  standard  data 
representations  for  communication  with  other  systems?  DC. 1(1) 

H 

N 

VII.  INSPECTOR'S  COMMENTS 

Make  any  general  or  specific  comments  that  relate  to  the  quality  observed  while 
applying  this  checklist. 


Y 

N 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

Y 

N 
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METRIC  WORKSHEET  2a 
DESIGN/SYSTEM  LEVEL 


DATE;  _ 

INSPECTOR: 


I.  COMPLETENESS  (CORRECTNESS,  RELIABILITY) 


Is  there  a  matrix  relating  itemized  requirements  to  modules  which  implement 
those  requirements?  TR.l 

How  many  major  functions  (CPCIS)  are  identified?  CP- 1 
How  many  functions  identified  are  not  defined?  CP. 1(2) 

How  many  defined  functions  are  not  used?  CP. 1(3) 

How  many  interfaces  between  functions  are  not  defined?  CP. 1(6) 

Number  of  total  problem  reports  recorded?  CP. 1(7) 

Number  of  those  reports  that  have  not  been  closed  (resolved?)  CP. 1(7) 

Profile  of  problem  reports:  (number  of  following  types)  a.  Computational 
__________________________________  b.  Logic 


II.  PRECISION  (RELIABILITY) 

Have  math  library  routines  to  be  used  been 
checked  for  sufficiency  with  regards  to 
accuracy  requirements?  AY. 1(3) 

Is  concurrent  processing  centrally 
controlled?  ET.l(l) 

How  many  error  conditions  are  reported 
by  the  system?  ET . 1 ( 2 ) 

How  many  of  those  errors  are  automatically 
fixed  or  bypassed  and  processing  continues? 
How  many,  require  operator  intervention?  ' 
Are  provisions  for  recovery  from  hardware' 
faults  provided?  ET.4(2) 

Are  provisions  for  recovery  from  device 
errors  provided?  ET.5(2) 


PORTABILITY,  REUSABILITY,  INTEROPERABILITY) 


Y 

N 

Y 

N 

Y 

N 

r 

-< 

N 

>- 

N 

c.  Input/output 

d.  Data  handling 

e.  OS/System  Support 

f.  Configuration 

g.  Routine/Routine 

interface 

b.  Routine/System 
Interface 

i.  Tape  Processing 

j.  User  interface 

k.  data  base  interface 

l .  user  requested 

changes 

m.  Preset  data 

n.  Global  variable 

definition 

p.  Recurrent  errors 

q.  Documentation 

r.  Requirement 

compliance 

s .  Operator 

t.  Questions 

u .  Hardware 


Is  a  hierarchy  of  system,  identifying  all  modules  in  the  system  provided? 
Number  of  Modules  SI.  1(2)  SI .1(1) 

Are  there  any  duplicate  functions?  SI. 1(2) 

Based  on  hierarchy  or  a  call /cal led  matrix,  how  many  modules  are  called  by 
more  than  one  other  module?  GE.l 

Are  the  constants  used  in  the  system  defined  once?  GE.2(5) 


METRIC  WORKSHEET  2a 
DESIGN/.SYSTEM  LEVEL 


SYSTEM 

NAME 


DATE; _ 

INSPECTOR: 


1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

9. 


1. 

2. 

3. 

4. 


1. 

2. 

3. 

4. 

5. 

6. 

7. 


1. 

2. 


.  1.  ...  1 

IV.  OPTIMIZATION  (EFFICIENCY) 

Are  storage  requirements  allocated  to  design?  SE.l(l) 

Are  virtual  storage  facilities  used?  SE.1(2) 

Is  dynamic  memory  management  used?  SE.1(5) 

Is  a  performance  optimizing  compiler  used?  SE.1(7) 

Is  global  data  defined  once?  CS.2(3) 

Have  Data  Base  or  files  been  organized  for  efficient  processing?  EE. 3(5) 
Is  data  packing  used?  EE. 2(5) 

Number  of  overlays  EE. 2(4) 

Overlay  efficiency  -  memory  allocation  EE. 2(4) 
max  overlay  size 
min  overlay  size 

Y 

n 

Y 

mm 

Y 

m 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

V.  SECURITY  (INTEGRITY) 

Are  user  Input/Output  access  controls  provided?  AC. 1(1) 

Are  Data  Base  access  controls  provided?  AC. 1(2) 

Is  memory  protection  across  tasks  provided?  AC. 1(3) 

Are  there  provisions  for  recording  and  reporting  errors?  AC. 2(1, 2) 

PM 

N 

WM 

N 

N 

wm 

N 

■  ■ 

VI.  SYSTEM  INTERFACES  (INTEROPERABILITY) 

How  many  other  systems  will  this  system  interface  with?  CC.l(l) 

Have  protoc  1  standards  been  established?  CC .1(2) 

Are  they  being  complied  with?  CC.l(Z) 

Number  of  modules  used  for  input  and  output  to  other  systems?  CC. 1(3,4) 
Has  a  standard  data  representation  been  established  or  translation 
standards  between  representations  been  established?  DC. 1(1) 

Are  they  being  complied  with?  DC. 1(2) 

Number  of  modules  used  to  perform  translations?  DC. 1(3) 

Y 

N 

Y 

N 

Y 

N 

Y 

N 

VII.  HUMAN  INTERFACE  (USABILITY) 

Are  all  steps  in  operation  described  including  alternative  flows?  OP. 1(1) 
Number  of  operator  actions?  OP. 1(4) 

Y  1  N 

■  ■  w 
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METRIC  WORKSHEET  2a 
DESIGN/SYSTEM  LEVEL 


DATE: 

INSPECTOR 


VII.  HUMAN  INTERFACE  (USABILITY)  Continued 


Estimated  or  Actual  time  to  perform?  OP. 1(4) 

Budgeted  time  for  complete  job?  OP. 1(4} 

Are  job  set  up  and  tear  down  procedures  described?  OP. 1(5) 

Is  a  hard  copy  of  operator  interactions  to  be  maintained?  OP. 1(6) 

Number  of  operator  messages  and  responses?  OP. 1(2) 

Number  of  different  formats?  OP. 1(2) 

Are  all  error  conditions  and  responses  appropriately  described?  OP. 1(2) 
Does  the  capability  exist  for  the  operator  to  interrupt,  obtain  status, 
save,  modify,  and  continue  processing?  OP.  1(3) 

Are  lesson  plans/training  materials  for  operators,  end  users,  and 
maintainers  provided?  TN.l(l) 

Are  realistic,  simulated  exercises  provided?  TN.1(2) 

Are  help  and  diagnostic  information  available?  TN.1(3) 

Number  of  input  formats  CM. 1(2) 

Number  of  input  values  CM.l(l) 

Number  of  default  values  CM .1(1) 

Number  of  self-identifying  input  values  CM. 1(3) 

Can  input  be  verified  by  user  prior  to  execution?  CM. 1(4) 

Is  input  terminated  by  explicitly  defined  by  logical  end  of  input?  CM. 1(5 
Can  input  be  specified  from  different  media?  CM. 1(6) 

Are  there  selective  output  controls?  CM. 2(1) 

Do  outputs  have  unique  descriptive  user  oriented  labels?  CM. 2(5) 

Do  outputs  have  user  oriented  units?  CM. 2(3) 

Number  of  output  formats?  CM. 2(4) 

Are  logical  groups  of  output  separated  for  user  emamination?  CM. 2(5) 

Are  relationships  between  error  messages  and  outputs  unambiguous?  CM. 2(6 
Are  there  provisions  for  directing  output  to  different  media?  CM. 2(7) 


VIIl-  TESTING  (TESTABILITY)  APPLY  TO  TEST  PLAN,  PROCEDURES,  RESULTS 


1.  Number  of  paths?  IN.l(l) 

2.  Number  of  paths  to  be  test^?^ , 

3.  Number  of  input  parameters ‘ 


4.  Number  of  input  parameters  to 
be  tested?  IN. 1(2) 

5.  Number  of  interfaces?  IN. 2(1) 


METRIC  WORKSHEET  2a 
OESIGN/SYSTEM  LEVEL 


DATE: 

INSPECTOR; 


VIII.  TESTING  (TESTABILITY)  -  APPLY  TO  TEST  PLAN,  PROCEDURES,  RESULTS  (CONTINUED) 


6.  Number  of  interfaces  to  be  tested?  IN. 2(1) 

7.  Number  of  itemized  performance 

8.  Number  of  performance  requirements  to  be 
tested?  IN. 2(2) 


9.  Number  of  modules?  IN. 3(1) 

10.  Number  of  modules  to  be 
exercised?  IN.  3(1) 

11.  Are  test  inputs  and  outputs 
provided  in  suiwnary  form? 


1.  Number  of  unique  data  items  in  data  base  SI. 1(6) 

2.  Number  of  preset  data  items  SI. 1(6)' 

3.  Number  of  major  segments  (files)  in  data  base  SI. 1(7) 


X  INSPECTOR'S  COMMENTS 


Make  any  general  or  specific  comments  about  the  quality  observed  while  applying  this 
checklist. 
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METRIC  WORKSHEET  2b 
DESIGN/MOOULE  LEVEL 


SYSTEM  NAME: 
MODULE  NAME: 


DATE:  _ 

INSPECTOR: 


I.  COMPLETENESS  (CORRECTNESS,  RELIABILITY) 

1.  Cari  you  clearly  distinguish  inputs,  outputs,  and  the  function  being  performed?  CP. 1(1)  y  N 

2.  How  many  data  references  are  not  defined,  computed,  or  obtained  from  an  external 

source?  CP. 1(2)  - 


3.  Are  all  conditions  and  processing  defined  for  each  decision  point?  CP. 1(5) 

4.  How  many  problem  reports  have  been  recorded  for  this  module?  CP. 1(7) 


5.  Profile  of  Problem  Reports: 

6.  Number  of  problem  reports  still  outstanding  C P.l( 7 ^ 

II.  PRECISION  (RELIABILITY) 

1.  When  an  error  condition  is  detected,  is  it 
passed  to  calling  module?  ET. 1(3) 

2.  Have  numerical  techniques  being  used  in  algori¬ 
thm  been  analyzed  with  regards  to  accuracy 
requirements?  AY. 1(4) 

3.  Are  values  of  inputs  range  tested?  ET. 2(2 ) 

4.  Are  conflicting  requests  and  illegal  combina¬ 
tions  identified  and  checked?  ET.2(3) 


5.  Is  there  a  check  to  see  if  all  necessary  data 
is  available  before  processing  begins?  ET. 2(5) 


6.  Is  all  input  checked,  reporting  all  errors, 
before  processing  begins?  ET .2(4) 


range  tested  before  use?  ET.3(2) 

8.  Are  subscripts  range  tested  before  use?  ET.3(3) 

9.  Are  outputs  checked  for  reasonableness  before 
processing  continues?  ET . 3(4) 


III.  STRUCTURE  (RELIABILITY,  MAINTAINABILITY,  TESTABILITY) 


E 

3 

Y 

N 

Y 

N 

Y 

z 

n 

3 

rz 

Y 

N.. 

Y 

N 

Y 

N 

Y 

N 

a.  Computational 

b.  Logic 

c.  Input/Output 

e.  System/OS  Support 

f.  Configuration 

g.  Routine/Routine  Inter¬ 
face 

h.  Routine/System  Inter¬ 
face 

i.  Tape  Processing 

j.  User  Interface 

k.  Data  Base  Interface 

l.  User  Requested  Changes 

m.  Preset  Data 

n.  Global  Variable  Defi¬ 
nition 

p.  Recurrent  Errors 

q.  Documentation 

r.  Requirement  Compliance 

t.  Operator 

u.  Questions 

v.  Hardware 


1.  How  many  Decision  Points  are  there? 

SI. 3 

2.  How  many  subdecision  Points  are 
there?  SI.  3 


3.  ^iow  many  conditional  branches  are 

there?  SI. 3 

4.  How  many  unconditional  branches 
are  there?  SI.3 


METRIC  WORKSHEET  2B 

DESIGN/MODULE  LEVEL 

SYSTEM  NAME: 

MODULE  NAME: 

Pg.  2 

III.  STRUCTURE  (RELIABILITY,  MAINTAINABILITY,  TESTABILITY)  (CONTINUED) 

1 

N 

7. 

Are  any  limitations  of  the  proces¬ 
sing  performed  by  the  module 
identified?  EX. 2(1) 

ifim 

8. 

Number  of  entrances  into  m^du|^|^ 

.. — j 

9. 

Number  of  exits  from  module  ,  . 

SI. 1(5) 

5.  Is  the  module  dependent  on  the 
source  of  the  input  or  the 
destination  of  the  output?  SI. 1(3) 

6.  Is  the  module  dependent  on  know¬ 
ledge  of  prior  processing?  SI. 1(3) 


IV.  REFERENCES  (MAINTAINABILITY,  FLEXIBILITY,  TESTABILITY,  PORTABILITY,  REUSABILITY, 
INTEROPERABILITY) 


1.  Number  of  references  to  system 
library  routines,  utilities  or  other 
system  provided  facilities  SS.l(l) 

2.  Number  of  input/output  actions 

MI. 1(2) 

3.  Number  of  calling  sequence  parameters 

MO. 2(3) 

4.  How  many  calling  sequence  parameters 
are  control  variables  MO. 2(3) 

5.  Is  input  passed  as  calling  sequence 
parameters  MO. 2(4) 

6.  Is  output  passed  back  to  calling 
module?  MO. 2(5) 

7.  Is  control  returned  to  calling 
module  MO. 2(6) 


8.  Is  temporary  storage  shared  with 
other  modules?  MO. 2(7) 

9.  Does  the  module  mix  input,  out¬ 
put  and  processing  functions  in 
same  module?  GE . 2 ( 1 ) 

10.  Number  of  machine  dependent 
functions  performed?  GE. 2(2) 

11.  Is  processing  data  volume  limited 

GE.2(3) 

12.  Is  processing  data  value  limited? 

JL  s  GE.2(4) 

13.  Is  a  common,  standard  subset  of 

„  programming  language  to  be  used? 

SS.  1  (2) 

N  14.  Is  the  programming  language 
*  available  in  other  machines? 

MI. 1(1) 


V.  EXPANDABILITY  (FLEXIBILITY) 


1.  Is  logical  processing  independent  of  storage  specification?  EX. 1(1) 

2.  Are  accuracy,  convergence,  or  timing  attributes  parametric?  EX. 2(1) 

3.  Is  module  table  driven?  EX. 2(2) 


VI.  OPTIMIZATION  (EFFICIENCY) 


1.  Are  specific  performance  requirements  (storage  and  routine)  allocated  to  this 
module?  EE.l 


METRIC  WORKSHEET  2b 

DESIGN/MODULE  LEVEL 

SYSTEM  NAME: 

MODULE  NAME: 

Pg.  3 

VI.  OPTIMIZATION  (EFFICIENCY)  (CONTINUED) 

2.  Which  category  does  processing  fall  in:  EE. 2 

Real-time 
On-1 ine 

Time-constrained 
Non-time  critical 

3.  Are  non-loop  dependent  functions  kept  out  of  loops?  EE. 2(1) 

4.  Is  bit/byte  packing/unpacking  performed  in  loops?  EE. 2(5) 

5.  Is  data  indexed  or  reference  efficiently?  EE. 3(5) 


VII.  FUNCTIONAL  CATEGORIZATION 


Categorize  function  performed  by  this  module  according  to  following: 


CONTROL  -  an  executive  module  whose  prime  function  is  to  invoke  other  modules. 


INPUT/OUTPUT  -  a  module  whose  prime  function  is  to  communicate  data  between 
- the  computer  and  the  user. 

PRE/POSTPROCESSOR  -  a  module  whose  prime  function  is  to  prepare  data  for  or 

- after  the  invocation  of  a  computation  or  data  management 

module. 


ALGORITHM  -  a  module  whose  prime  function  is  computation. 


DATA  MANAGEMENT  -  a  moudle  whose  prime  function  is  to  control  the  flow  of 
- ■  data  within  the  computer. 

SYSTEM  -  a  module  whose  function  is  the  scheduling  of  system  resources  for 
- other  modules. 


VIII.  CONSISTENCY 


1.  Does  the  design  representation  comply  with  established  standards  CS .1(1) 

2.  Do  input/output  references  comply  with  established  standards  CS .1(3) 

3.  Do  calling  sequences  comply  with  established  standards  CS .1(2) 

4.  Is  error  handling  done  according  to  established  standards  CS. 1(4) 

5.  Are  variable  named  according  to  established  standards  CS .2(2) 

6.  Are  global  variables  used  as  defined  globally  CS.2(3) 
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METRIC  WORKSHEET  2b 
CESIGN/MODULE  LEVEL 


SYSTEM  NAME: 
MODULE  NAME: 


Pg.  4 


IX.  INSPECTOR'S  COMMENTS 


Make  any  specific  or  general  comments  about  the  quality  observed  while  applying  this 
checklist? 
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METRIC  WORKSHEET  3 
SOURCE  CODE/MODULE  LEVEL 


SYSTEM  NAME: 
MODULE  NAME: 


DATE:  _ 

INSPECTOR: 


I.  STRUCTURE  (RELIABILITY,  MAINTAINABILITY,  TESTABILITY) 


1.  Number  of  lines  of  code  MO. 2(2) 

2.  Number  of  lines  excluding  comments 

SI. 4(2) 

3.  Number  of  machine  level  language 
statements  SD. 3(1) 

4.  Number  of  declarative  statements 

SI. 4 

5.  Number  of  data  manipulation  state¬ 
ments  SI. 4 

6.  Number  of  statement  labels  SI. 4(6) 
(Do  not  count  format  statements) 

7.  Number  of  entrances  into  module 

SI. 1(5) 

8.  Number  of  exits  from  module 

SKI  (5) 

9.  Maximum  nesting  level  SI. 4(7) 

10.  Number  of  decision  points 

(IF,  WHILE,  REPEAT,  DO,  CASE)  SI. 3 


11.  Number  of  sub-decision  points 

SI. 3 

12.  Number  of  conditional  branches 
(computed  go  to)  SI. 4(8) 

13.  Number  of  unconditional  branches 
(GOTO,  ESCAPE)  SI. 4(9) 

14.  Number  of  loops  (WHILE,  DO) 

SI. 4(3) 

15.  Number  of  loops  with  jumps  out  of 
loop  SI. 4 (3) 

16.  Number  of  loop  indicies  that  are 
modified  SI. 4(4) 

17.  Number  of  constructs  that  perform 
module  modification  (SWITCH, 

ALTER)  SI. 4(5) 

18.  Number  of  negative  or  complicated 
compound  boolean  expressions 

19.  Is  a  structured  lanquaqe  usecl^ 

SI. 2 

20.  Is  flow  top  to  bottom  (are  there 
any  backward  branching  G0T0s)$j  4^ 


II.  CONCISENESS  (MAINTAINABILITY)  -  SEE  SUPPLEMENT 


1.  Number  of  operators  C0.1 

2.  Number  of  unique  operators  C0.1 


3.  Number  of  Operands  C0.1 

4.  Number  of  unique  operands  C0.1 


III.  SELF-DESCRIPTIVENESS  (MAINTAINABILITY,  FLEXIBILITY,  TESTABILITY,  PORTABILITY,  REUSABILITY) 


1.  Number  of  lines  of  comments  SD.l 

2.  Number  of  non-blank  lines  of  comments 

SD.l 

3.  Are  there  prologue  comments  provided 
containing  information  about  the 
function,  author,  version  number, 
date,  inputs,  outputs,  assumptions 
and  limitations? 

4.  Is  there  a  comment  which  indicates 
what  itemized  requirement  is 
satisfied  by  this  module?  SD. 2(1 ) 

5.  How  many  decision  points  and  trans¬ 
fers  of  control  are  not  commented? 

SD.2(3) 

6.  Is  all  machine  language  code  com¬ 
mented?  SD.2(4) 


7.  Are  non-standard  HOL  statements 
commented?  SD. 2(5) 


8.  How  many  declared  variables  are 
not  described  by  comments? 

SD. 2(6) 

9.  Are  variable  names  (mnemonics) 
descriptive  of  the  physical  or 
functional  property  they 
represent?  SD . 3(2; 

10.  Do  the  comments  do  more  than 
repeat  the  operation?  SD. 2(7) 


11.  Is  the  code  logically  blocked  and 
indented?  SD. 3( 3) 


12.  Number  of  lines  with  more  than 
1  statement.  SD. 3(4) 

13.  Number  of  continuation  lines 


METRIC  WORKSHEET  3 

SOURCE  CODE  MODULE  LEVEL 

SYSTEM  NAME: 

MODULE  NAME: 

Pg.  2 

IV.  INPUT/OUTPUT  (RELIABILITY,  FLEXIBILITY,  PORTABILITY) 

1.  Number  of  input  statements  MI. 1(2) 

4. 

Are  inputs  range-tested  (for 

inputs  via  calling  sequences. 

2.  Number  of  output  statements  MI. 1(2) 

global  data,  and  input 

statements)  ET.2(2) 

3.  Is  amount  of  input  that  can  be 

5. 

Are  possible  conflicts  or  illegal 

handled  parametric?  G£.2{3)  - - 

combinations  in  inputs  checked? 

Y  N 

ET.2(3) 

6. 

Is  there  a  check  to  determine  if 

all  data  is  available  prior  to 

processing?  ET.2(5) 

V.  REFERENCES  (RELIABILITY,  MAINTAINABILITY.  TESTABILITY.  FLEXIBILITY.  PORTABILITY,  REUSABILITY) 


1.  Number  of  calls  to  other  modules 

MO. 2(1) 

2.  Number  of  references  to  system 
library  routines,  utilities,  or 
other  system  provided  functions 

SS.l(l) 

3.  Number  of  calling  sequence  parameters 

MO. 2(3) 

4.  How  many  elements  in  calling 
sequences  are  not  oarameters? 

MO. 2(3) 

5.  How  many  of  the  calling  parameters 
(input)  are  control  variables? 

MO. 2(3) 


6.  How  many  parameters  passed  to  or 
from  other  modules  are  not  defined 
in  this  module?  MO. 2(3) 

7.  Is  input  data  passed  as  parameter? 

MO. 2(4) 


8.  Is  output  data  passed  back  to 
calling  module?  MO. 2(5) 


9.  Is  control  returned  to  calling 
module?  MO. 2(6) 


VI.  DATA  (CORRECTNESS,  RELIABILITY,  MAINTAINABILITY,  TESTABILITY) 


1. 

Number  of  local 

variables 

SI. 4(10) 

2. 

Number  of  global 

variables 

SI. 4(10) 

3. 

Number  of  global 

variables 

renamed 

SE.l (3) 

VII.  ERROR  HANDLING  -  (RELIABILITY) 


1.  How  many  loop  and  multiple  transfer 
index  parameters  are  not  range 
tested  before  use?  ET.3(2) 

2.  Are  subscript  values  range  tested 
before  use?  ET .  3(3) 

3.  When  an  error  condition  occurs,  is  it 
passed  to  the  calling  module?  E T . 1 ( 3) 

4.  Are  the  results  of  a  computation 
checked  before  outputting  or  before 
processing  continues?  ET . 3(4 ) 


4.  How  many  global  variables  are  not 
used  consistently  with  respect  to 
units  or  type?  CS. 2(4 ) 

5.  How  many  variables  are  used  for 
more  than  one  purpose?  CS.2(3) 


VIII.  (EFFICIENCY) 


1.  Number  of  mix  mode  expressions? 

EE. 3(3) 

2.  How  many  variables  are  initialized 
when  declared?  EE. 3(2) 

3.  How  many  loops  have  non- loop 
dependent  statements  in  them? EE. 20 

4.  How  many  loops  have  b it/by te 
packing/unpacking?  EE. 2(5) 

SE.l (6) 

5.  How  many  compound  expressions 
defined  more  than  once?  EE. 2(3) 
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METRIC  WORKSHEET  3 
SOURCE  CODE/MODULE  LEVEL 


SYSTEM  NAME: 
MODULE  NAME: 


IX.  PORTABILITY 


1 .  Is  code  independent  of  word  and 
character  size?  MI. 1(3) 


2.  Number  of  lines  of  machine  language 
statements.  MI.l 

3.  Is  data  representation  machine 
independent?  MI.l (4) 


4.  Is  data  access/storage  system  soft¬ 
ware  independent?  SS.l 


.  FLEXIBILITY 


1.  Is  module  table  driven  EX. 2(2) 

2.  Are  there  any  limits  to  data 
values  that  can  be  processed? 

GE.2(4) 

3.  Are  there  any  limits  to  amounts 
of  data  that  can  be  processed? 

GE.2(3) 

4.  Are  accuracy,  convergence  and 
timing  attributes  parametric? 

EX. 2(1 ) 


XI.  DYNAMIC  MEASUREMENTS  (EFFICIENCY,  RELIABILITY) 


1.  During  execution  are  outputs  within  accuracy  tolerances?  AY. 1(5) 

2.  During  module/development  testing,  what  was  run  time?  EX. 2(3) 

3.  Complete  memory  map  for  execution  of  this  module  SE. 1(4) 


APPLICATION 

SYSTEM 

DATA 

OTHER 

4.  During  execution  how  many  data  items  were  referenced  but  not  modified  EE. 3(6) 

5.  During  execution  how  many  data  items  were  modified  EE. 3(7) 


XII.  INSPECTORS  COMMENTS 

Make  any  general  or  specific  comments  that  relate  to  the  quality  observed  by  you  while 
applying  this  checklist: 
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3.4  TECHNIQUES  FOR  APPLYING  MEASUREMENTS 

Section  1.5  identified  organizational  approaches  for  utilizing  the  quality 
metric  concepts  during  a  software  development.  These  approaches  included 
both  acquisition  environments  and  internal  development  environments.  The 
purpose  of  this  section  is  to  describe,  at  a  lower  level,  how  the  metrics 
would  be  applied  in  either  case. 

The  first  technique  for  applying  the  metrics  is  by  formal  inspection.  The 
formal  inspection  is  performed  by  personnel  of  an  organization  independent 
of  the  development  organization  (the  acquisition  office,  an  independent  quality 
assurance  group,  or  an  independent  contractor).  The  metric  worksheets  are 
applied  to  delivered  products  at  scheduled  times  and  the  results  are  formally 
reported. 

The  second  technique  is  to  utilize  the  worksheets  during  structured  design 
and  code  walkthroughs  held  by  the  development  team.  A  specific  participant 
of  the  walkthrough  can  be  designated  for  applying  the  worksheets  and  report¬ 
ing  any  deficiencies  during  the  walkthrough  or  a  representative  of  the 
quality  assurance  organization  can  participate  in  the  walkthrough  with  the 
purpose  of  taking  the  measurements  of  the  design  or  code. 

The  last  technique  is  for  the  development  team  to  utilize  the  worksheets  as 
guidelines,  self-evaluations  or  in  a  peer  review  mode  to  enhance  the  qualtiy 
of  the  products  they  produce. 


SECTION  4 


PROCEDURES  FOR  ASSESSING  THE  QUALITY  OF  THE  SOFTWARE  PRODUCT 

4.1  INTRODUCTION 

The  benefits  of  applying  the  software  quality  metrics  are  realized  when  the 
information  gained  from  their  application  is  analyzed.  The  analyses  that  can 
be  done  based  on  the  metric  data  are  described  in  the  subsequent  paragraphs. 
There  are  three  levels  at  which  analyses  can  be  performed.  These  levels  are 
related  to  the  level  of  detail  to  which  the  quality  assurance  organization 
wishes  to  go  in  order  to  arrive  at  a  quality  assessment. 

4.2  INSPECTOR'S  ASSESSMENT 

The  first  level  at  which  an  assessment  can  be  made  relies  on  the  discipline 
and  consistency  introduced  by  the  application  of  the  worksheets.  An  inspec¬ 
tor,  using  the  worksheets,  asks  the  same  questions  and  takes  the  same  counts  for 
each  module's  source  code  or  design  document, etc.  that  is  reviewed.  Based  on 
this  consistent  evaluation,  a  subjective  comparison  of  products  can  be  made. 

la.  Vocumnt  Tjupe.eAon' Aaaeaamejit 

The  last  section  in  each  worksheet  is  a  space  for  the  inspec¬ 
tor  to  make  comments  on  the  quality  observed  while  applying  the 
worksheet.  Comments  should  indicate  an  overall  assessment  as  well 
as  point  out  particular  problem  areas  such  as  lack  of  comments, 
inefficiencies  in  implementation,  dr  overly  complex  control  flow. 

lb.  Compete.  A Aammenta  Sy&tum  RevcenJ 

By  compiling  all  of  the  inspector's  assessments  on  the  various 
documents  and  source  code  available  at  any  time  during  the  develop¬ 
ment,  deficiencies  can  be  identified. 

4.3  SENSITIVITY  ANALYSIS 

The  second  level  of  detail  utilizes  experience  gained  through  the  application 
of  metrics  and  the  accumulation  of  historical  information  to  take  advantage 
of  the  quantitative  nature  of  the  metrics.  The  values  of  the  measurements 
are  used  as  indicators  for  evaluation  of  the  progress  toward  a  high  quality 
product. 


At  appropriate  times  during  a  large-scale  development,  the  application  of 
the  worksheets  allows  calculation  of  the  metrics.  The  correspondence  of  the 
worksheets  to  the  metrics  is  shown  in  Appendix  B  of  [MCCA79].  The  results  of 
these  calculations  is  a  matrix  of  measurements.  The  metrics  that  have  been 
established  to  date  are  at  two  levels-system  level  and  module  level.  The 
approach  to  be  described  is  applicable  to  both  levels  and  will  be  described 
in  relationship  to  the  module  level  metric. 

A  n  by  k  matrix  of  measurements  results  from  the  application  of  the  metrics  to 
the  existing  products  of  the  development  (e.g.,  at  design,  the  products  might 
include  review  material,  design  specifications,  test  plans,  etc.)  where  there 
are  k  modules  and  n  module  level  measurements  applicable  at  this  particular  time. 

"’ll  ffl12  .  .  .  .  mlk 

m21 

mnl  '  mnk 

This  matrix  represents  a  profile  of  all  of  the  modules  in  the  system  with 
respect  to  a  number  of  characteristics  measured  by  the  metrics.  The  analyses 
that  can  be  performed  are  described  in  the  following  steps: 

2a.  Aiawa  VcviicuUon  HeaiuA.mmti 

Each  row  in  the  above  matrix  represents  how  each  module  in  the 
system  scored  with  respect  to  a  particular  metric.  By  summing  all 
the  values  and  calculating  the  average  and  standard  deviation  for 
that  metric,  each  individual  module's  score  can  then  be  compared 
with  the  average  and  standard  deviation.  Those  modules  that 
score  less  than  one  standard  deviation  from  the  average  should  be 
identified  for  further  examination.  These  calculations  are 
illustrated  below: 

k 

for  metric  1i  Average  Score  =  Ai  =  z  M.  ./k 

([  £ 

Standard  Deviation  =  o.  =  I  /k 

i  j=1  1J  i 

Report  Module  j  if  M^  <  -  0i 
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2b.  A 44£A4  Low  Sy&tem  Scoma 

In  examining  a  particular  measure  across  all  modules,  consistently 
low  scores  may  exist.  It  may  be  that  a  design  or  Implementation 
technique  used  widely  by  the  development  team  was  the  cause.  This 
situation  Identifies  the  need  for  a  new  standard  or  stricter 
enforcement  of  existing  standards  to  Improve  the  overall  develop¬ 
ment  effort. 

2c.  Aaaeaa  Soot tea  Agcun&t  ThAe^koldi 

As  experience  is  gained  with  the  metrics  and  data  is  accumulated, 
threshold  values  or  Industry  acceptable  limits  may  be  established. 
The  scores,  for  each  module  for  a  particular  metric  should  be  com¬ 
pared  with  the  established  threshold.  A  simple  example  is  the 
percent  of  comments  per  line  of  source  code.  Certainly  code 
which  exhibits  only  one  or  two  percent  measurements  for  this 
metric  would  be  Identified  for  corrective  action.  It  may  be 
that  ten  percent  Is  a  minimum  acceptable  level.  Another  example 
Is  the  complexity  measure.  A  specific  value  of  the  complexity 
measure  greater  than  some  chosen  value  should  be  identified  for 
corrective  action 


Report  Module  j  if  M^  <T,  (or>T  for  complexity 

J  measures) 

Where  Ti  =  threshold  value 

specified  for  metric  i. 

4.4  USE  OF  NORMALIZATION  FUNCTION  TO  ASSESS  QUALITY 

The  last  level  of  assessing  Quality  is  using  the  normalization  functions  to 
predict  the  quality  in  quantitative  terms.  The  normalization  functions 
are  utilized  in  the  following  manner. 

For  a  particular  time  there  is  an  associated  matrix  of  coefficients  which 
represent  the  results  of  linear  multivariate  regression  analyses  against 
empirical  data  (past  software  developments).  These  coefficients,  when 
multiplied  by  the  measurement  matrix  results  in  an  evaluation  (prediction) 
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of  the  quality  of  the  product  based  on  the  development  to  date.  This 
coefficient  matrix,  shown  below,  has  n  columns  for  the  coefficients  of 
the  various  metrics  and  11  rows  for  the  11  quality  factors. 

rCll  C12  .  .  .  cln 


Lcll,l  cll,nJ 


To  evaluate  the  current  degree  or  level  of  a  particular  quality  factor,  i, 
for  a  module,  j,  the  particular  column  in  the  measurement  matrix  is  multiplied 
by  the  row  in  the  coefficient  matrix.  The  resultant  value: 


Ci,l 


mi ,  j  +  ci  ,2 


2 » J  i  m,j 

is  the  current  predicted  rating  of  that  module,  j  for  the  quality  factor,  i. 
This  predicted  rating  is  then  compared  to  the  previously  established  rating 
to  determine  if  the  quality  is  as  least  as  sufficient  as  required.  The 
coefficient  matrix  should  be  relatively  sparse  (many  =  0).  Only  sub¬ 
sets  of  the  entire  set  of  metrics  applicable  at  any  one  time  relate  to  the 
criteria  of  any  particular  quality  factor. 


”  ri,J 


Multiplying  the  complete  measurement  matrix  by  the  coefficient  matrix 
results  in  a  ratings  matrix.  This  matrix  contains  the  current  predicted 
ratings  of  each  module  for  each  quality  factor.  Each  module  then  can  be 
compared  with  the  preset  rating  for  each  quality  factor. 

rrn  r12  •  rl,k  1 


cm=rJ= 


n,i 


rll ,k J 


This  approach  represents  the  most  formal  approach  to  evaluating  the  quality 
of  a  product  utilizing  the  software  quality  metrics.  Because  the  coefficient 
matrix  has  been  developed  only  for  a  limited  sample  in  a  particular  environment, 
it  Is  neither  generally  applicable  nor  has  statistical  confidence  in  its  value 
been  achieved. 


55 


To  use  the  normalization  functions  that  currently  exist  the  following  steps 
should  be  performed. 


3a.  A pply  Normalization  Function 

Table  4.4-1  contains  the  normalization  functions  that  currently 
exist.  If  any  of  the  quality  factors  identified  in  that  table 
have  been  specified  as  a  requirement  of  the  development,  then  the 
metrics  identified  in  the  table  should  be  substituted  into  the 
equation  and  the  predicted  rating  calculated.  Normalization 
functions  which  Include  several  metrics  can  be  used  if  available, 
otherwise  functions  for  individual  metrics  should  be  used.  This 
predicted  rating  should  be  compared  with  the  specified  rating. 

To  illustrate  the  procedure  the  normalization  function  that  has 
been  developed  for  the  factor  Flexibility  will  be  used.  The 
normalization  function,  applicable  during  the  design  phase, 
relates  measures  of  modular  implementation  to  the  flexibility  of 
the  software.  The  predicted  rating  of  flexibility  is  in  terms 
of  the  average  time  to  implement  a  change  in  specifications.  The 
normalization  function  is  shown  in  Figure  4.4-1.  The  measurements 
associated  with  the  modular  implementation  metric  are  taken  from 
design  documents.  The  measurements  involve  identifying  if  input, 
output  and  processing  functions  are  mixed  in  the  same  module,  if 
application  and  machine-dependent  functions  are  mixed  in  the  same 
module  and  if  processing  is  data  volume  limited.  As  an  example, 
assume  the  measurements  were  applied  during  the  design  phase  and  a 
value  of  0.65  was  measured.  Inserting  this  value  in  the  normalization 
function  results  in  a  predicted  rating  for  flexibility  of  .33  as 
identified  by  point  A  in  Figure  4.4-1.  If  the  Acquisition  Manager 
had  specified  a  rating  of  0.2  which  is  Identified  by  point  B,  he 
has  an  indication  that  the  software  development  is  progressing  well 
with  respect  to  this  desired  quality. 
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An  organization  using  this  manual  is  encouraged  to  establish  these 
functions  in  its  specific  environment  by  following  the  procedures 
described  in  [MCCA77]  and  [MCCA79]. 


Table  4.4-1  Normalization  Functions 


RELIABILITY  (DESIGN) 

MULTIVARIATE  .18  MCT  .  +  .19  M_T  , 

FUNCTION  ET-'  SI-3 

ET.l  Error  Tolerance  Checklist 

SI.3  Complexity  Measure 

1 

RELIABILITY  (IMPLEMENTATION) 

MULTIVARIATE  .48MFT  ,+  .14MCT  , 

FUNCTION  M>l 

ET.l  Error  Tolerance  Checklist 

51.3  Complexity  Measure 

SI.l  Design  Structure  Measure 

51. 4  Coding  Simplicity  Measure 

INDIVIDUAL  .57  M  . 

.58  HsI1 

•53  "SI.3 
•»"SI.4 

MAINTAINABILITY  (DESIGN) 

INDIVIDUAL  .57  Mc.  , 

FUNCTION  „  „ 

•53  MSI.l 

SI.3  Complexity  Measure 

SI.l  Oeslgn  Structure  Measure 

MAINTAINABILITY  (IMPLEMENTATION) 

MULTIVARIATE  -.2+.61  MCT  ,+  .14MMn  ,+.33cn  , 
FUNCTION  51  • 3  "U*Z  SD*Z 

51.3  Complexity  Measure 

MO. 2  Modular  Implementation 

Measure 

SD.2  Effectiveness  of  Comments 

Measure 

SD  3  Descriptiveness  of 

Implementation 

Language  Measure 

SI.l  Design  Structure 

Measure 

51. 4  Coding  Simplicity  Measure 

INDIVIDUAL 

FUNCTIONS  2.1  M$I  3 

•71  MSD.2 
,6  MSD.3 
•5  MSI.l 
*4  MSI.4 

FLEXIBILITY  (DESIGN) 

INDIVIDUAL 

FUNCTIONS  .51  M^  2 

.be  mge;2 

MO. 2  Modular  Implementation 

GE.2  Generality  Checklist 

(Continued) 


V. 
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Table  4.4-1  Normalization  Functions  (Continued) 


FLEXIBILITY  (IMPLEMENTATION) 

MULTIVARIATE  ^M^  ,+  .44M 
FUNCTION 


GE.2 


+.09M 


SD.3 


INDIVIDUAL 

FUNCTIONS  .6^  2 

*72MGE.2 

*59MSD.2 

•56MSD.3 

MO. 2  Modular  Implementation 
Measure 

GE.2  Generality  Checklist 

SD.2  Effectiveness  of  Conments 
Measure 

SD.3  Descriptiveness  of 
Implementation 

Language  Measure 

PORTABILITY  (IMPLEMENTATION) 

MULTIVARIATE  -1.7+.19Mcn  ,+  .76Mcf,  ,+2.5Mcn  ,+.64^1,.  . 

FUNCTI0N  SO. 1  SO. 2  SD.3  WI.l 

INDIVIDUAL 

FUNCTIONS  1>07MSI  1 

1JMMI.i 

1,5MS0.2 

SD.l  Quantity  of  Comments 

SD.2  Effectiveness  of 

Comments  Measure 

SD.3  Descriptiveness  of 
Implementation 

Language  Measure 

MI .  1  Machine  Independence 
Measure 

SI . 1  Design  Structure 

Measure 

HO. 2  MODULAR  IMPLEMENTATION  MEASURE/OESIGN  PHASE 


3b.  Caicuiate  Confidence,  in  QuaLuty  Aaaea-i meni 


Using  statistical  techniques  a  level  of  confidence  can  be 
calculated.  The  calculation  is  based  on  the  standard  error  of 
estimate  for  the  normalization  function  (given  in  Table  3. 3. 3-2, 
Vol .  I)  and  can  be  derived  from  a  normal  curve  table  found  in  most 
statistics  texts.  An  example  of  the  devivation  process  is  shown 
in  Figure  4.4-2  for  the  situation  described  above.  Here  it  is 
shown  that  the  Acquisition  Manager  has  an  86  percent  level  of 
confidence  that  the  flexibility  of  the  system  will  be  better  than 
the  specified  rating. 


MEAN  *  .33 


LEVEL  OF  CONFIDENCE  =  Pr  jx  ». . 2 }  ■  . 36  (SHADED  AREA) 


Figure  4.4-2  Determination  of  Level  of  Confidence 
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4,5  REPORTING  ASSESSMENT  RESULTS 

Each  of  the  proceeding  steps  described  In  this  section  are  easily  automated. 
If  the  metrics  are  applied  automatically  then  the  metric  data  Is  available  In 
machine  readable  form.  If  the  worksheets  are  applied  manually,  then  the 
data  can  be  entered  Into  a  file,  used  to  calculate  the  metric,  and  formatted 
Into  the  measurement  matrix  format.  The  automation  of  the  analyses  involve 
simple  matrix  manipulations.  The  results  of  the  analyses  should  be  reported 
at  various  levels  of  detail.  The  formats  of  the  reports  are  left  to  the 
discretion  of  the  quality  assurance  organization.  The  content  of  the 
reports  to  the  different  managers  Is  recommended  In  the  following  paragraphs. 

la*  Repoat  to  the  Acqauttion  ManageA/Vevetopment  Manage A 

The  report  content  to  the  Acquisition  Manager  and  the  Develop¬ 
ment  manager  should  provide  summary  information  about  the  progress 
of  the  development  toward  the  quality  goals  identified  at  the  be 
beginning  of  the  project. 

For  example  If  ratings  were  specified  for  several  quality  factors, 
the  current  predicted  ratings  should  be  reported. 

PREDICTED  RATING 

QUALITY  GOALS  BASED  ON  DESIGN  DECUMENT 

RELIABILITY  .9  .8 

MAINTAINABILITY  .8  .95 

If  specific  ratings  were  not  identified  but  the  important  qualities 
were  Identified,  a  report  might  describe  the  percentage  of  modules 
that  currently  are  judged  to  be  below  the  average  quality  (as  a 
result  of  the  sensitivity  analysis)  or  that  are  below  a  specified 
threshold  value  its  a  result  of  the  threshold  analysis;.  These 
statistics  provide  a  progress  status  to  the  manager.  Further 
progress  status  Is  Indicated  by  reporting  the  quality  growth 
of  in*  system  or  of  individual  modules.  The  quality  growth  is 
awklwl  fc/  reporting  th#  scores  achieved  during  the  various 
abases  t*  development  ultimately  tn*  ratings  should  progressi vcl y 


score  higher  than  those  received  during  requirements.  This  progress 
is  based  on  the  identification  of  problems  in  the  early  phases 
which  can  then  be  corrected. 

lb.  Report*  to  QuaLLty  AiiuAance  Manage*. 

In  addition  to  the  summary  quality  progress  reports  described 
in  la,  the  quality  assurance  manager  and  his  staff  will  want 
detailed  metric  reports.  These  reports  will  provide  all  of 
the  results  of  the  Analyses  described  in  4.2,  4.3,  and  4.4,  and 
perhaps  provide  the  measurement  matrix  itself  for  examinations. 

In  addition  to  the  detailed  reports,  the  quality  assurance  manager 
should  be  provided  with  reports  on  the  status  of  the  application  of 
the  metrics  themselves  by  the  quality  assurance  staff.  These 
status  reports  will  provide  information  on  total  number  of  modules 
and  the  number  which  inspectors  have  analyzed. 

lc.  Report*  to  the  Development  Team 

The  development  team  should  be  provided  detailed  information  on 
an  exception  basis.  This  information  is  derived  from  the 
analyses.  Examples  of  the  information  would  be  quality  problems 
that  have  been  identified,  which  characteristics  or  measurements 
of  the  software  products  are  poor,  and  which  modules  have  been 
identified  as  requiring  rework.  These  exception  reports  should 
contain  the  details  of  why  the  assessment  revealed  them  as 
potential  problems.  It  is  based  onthis  information  that  corrective 
actions  will  be  taken. 
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